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Executive Summary 

This report presents the final results from an Applied Meteorology Unit (AMU) evaluation of the upgraded 
version of the Regional Atmospheric Modeling System (RAMS) Numerical Weather Prediction (NWP) model 
as run in the Eastern Range Dispersion Assessment System (ERDAS). ERDAS is designed to provide 
emergency response guidance to the 45th Space Wing/Eastem Range Safety (45 SW/SE) in support of 
operations at the Eastern Range in the event of an accidental hazardous material release or an aborted vehicle 
launch. 

ERDAS uses the RAMS NWP model to generate prognostic wind and temperature fields for input into 
ERDAS diffusion algorithms. In addition, RAMS predicts a number of other meteorological quantities on four 
nested grids with horizontal resolutions of 60, 15, 5, and 1.25 km, respectively. Since the 1.25-km grid is 
centered over the Kennedy Space Center (KSC) and Cape Canaveral Air Force Station (CCAFS), real-time 
RAMS forecasts provide an opportunity for improved weather forecasting in support of space operations 
through high-resolution NWP over the complex land-water interfaces of KSC/CCAFS. The 45 SW/SE and the 
45th Weather Squadron (45 WS) tasked the AMU to evaluate the capabilities and accuracy of RAMS for all 
seasons and under various weather regimes during 1 999 and 2000. 

The AMU subdivided the RAMS evaluation into three seasons, including the 1999 Florida warm season 
(May- August), the 1999-2000 cool season (November-March), and the 2000 warm season (May-September). 
Much of this final report focuses on the 1999-2000 cool and 2000 warm seasons since the ERDAS RAMS 
interim report summarized the results of the 1 999 warm season. 

The RAMS evaluation includes an objective and subjective component. The objective component involves 
point forecast error statistics at all available observational locations on grid 4(1 .25 km resolution), and selected 
observations on grids 1-3. The point error statistics that were examined include the Root Mean Square (RMS) 
error (total error), bias (systematic error component), and error standard deviation (random error component). 
The objective evaluation in this report consists of five segments for examining these point error statistics: 

• Verification of the operational RAMS for the 1999-2000 cool and 2000 warm seasons, 

• Surface wind regime classification for the 2000 warm season, 

• Thunderstorm day regime classification for the 2000 warm season, 

• Comparison of point error statistics between the operational configuration and a RAMS 
configuration with a coarser horizontal resolution, and 

• Comparison of RAMS errors to the Eta model errors at the Shuttle Landing Facility (station 
symbol TTS). 

The subjective component of the RAMS evaluation focused on the verification of fronts, precipitation 
across the Florida peninsula, and low-level temperature inversions at the Cape Canaveral rawinsonde during the 
1999-2000 cool season. The warm-season subjective evaluation focused on the verification of sea breezes 
during 1999 and 2000, precipitation on grid 4 during 2000, and thunderstorm initiation on grid 4 during 2000. 

The most notable point-forecast errors associated with the operational RAMS forecasts are as follows: 

• RAMS had a surface-based, daytime low-level cold temperature bias that occurs during all 
seasons, reaching a maximum of 4.5°C in the cool season and 3.5°C in the warm season. This cold 
bias is consistent with the results found in the ERDAS RAMS interim evaluation report. 

• The vertical temperature profile throughout the atmosphere was typically too stable during both 
seasons (e.g. too cold near the surface and too warm aloft by 0.5-1. 0°C); however, in the lowest 
0.5 km during the early morning hours, the RAMS temperature profile is too unstable (too warm at 
the surface by 0.5— 1 .0°C and too cold at 0.5 km by nearly 3°C). 

• A surface-based nocturnal moist bias was found during the cool season whereas a daytime dry bias 
occurred in the 2000 warm season. 

• At the KSC/CCAFS wind towers, wind direction RMS errors grew rapidly from 20° at 
initialization to 40° within the first 2 hours of model integration. 

• The largest wind direction RMS errors of 60-70° occurred at the surface during the late night and 
early morning hours associated with light and variable winds common during those times. 

• RAMS also experienced a slight low-level easterly wind bias and a southerly wind bias (maximum 
magnitude about 1-2 m s' 1 ) at all levels above the surface during both seasons. 
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The sea-breeze evaluation was examined for the operational RAMS configuration using data from the 1999 
and 2000 warm-seasons. In addition, the operational RAMS sea-breeze forecasts were compared to a coarser 
resolution, 3-grid configuration of RAMS and to the operational National Centers for Environmental Prediction 
(NCEP) Eta model. The results of these verifications are given below: 

• RAMS did an excellent job in forecasting the onset and movement of the east coast sea breeze 
(ECSB). The 1200 UTC forecast cycle exhibited the highest probability of detection (0.98) and 
best overall skill. 

• Despite the low-level cold temperature bias, RAMS demonstrated this high skill in predicting the 
occurrence of the ECSB because the cold temperature bias was prevalent over both land and water. 
As a result, the thermal contrast between land and water that drives the sea-breeze circulation was 
represented well by the model. 

• The RMS error in timing of the sea-breeze onset was between 1.5— 2.1 h at all towers and the bias 
was negligible. 

• In the 4-grid/3-grid sea-breeze comparison during 2000, the higher-resolution 4-grid configuration 
outperformed the 3 -grid forecasts in nearly all skill categories. 

• RAMS was more skillful than the Eta model for the 0000 UTC cycle only. 

• In the 1200 UTC forecast cycle, the RAMS probability of detection was significantly higher than 
the Eta model, but so was the false alarm rate and bias. The resulting improvement in skill scores 
of the RAMS over the Eta model were not statistically significant because RAMS tended to over- 
forecast the sea-breeze occurrence at TTS. 

• These results indicate that, despite the comparable or slightly better objective error statistics in the 

Eta model, the phenomenological verification of the ECSB improves over the Eta model when 

running the RAMS model with fine horizontal grid spacing such as in the current configuration. 

For the precipitation and thunderstorm initiation verifications, the AMU divided grid 4 into six zones, 3 
coastal and 3 inland. Precipitation and thunderstorm activity were verified each day during peak convective 
hours (1500-2300 UTC, or 1 100-1900 EDT). For the thunderstorm initiation verification, the AMU defined a 
RAMS thunderstorm based on a predicted minimum vertical velocity in the charge zone of a forecast storm, 
combined with forecast precipitation at the ground. The results of the precipitation verification indicate that: 

• RAMS predicted precipitation with the highest skill over the inland zones whereas the model had 
the poorest skill over the coastal zones, especially the southeastern zone of grid 4. 

• The 1200 UTC cycle was generally more skillful than the 0000 UTC forecasts. Based on the 1200 
UTC cycle, the most accurate precipitation forecasts occurred between 1600-2000 UTC and the 
least accurate forecasts occurred after 2000 UTC. The reduction of skill after 2000 UTC could be 
caused by the model’s inability to forecast adequately the evolution and interaction of 
thunderstorm outflow boundaries. 

The results of the thunderstorm initiation verification suggest the following: 

• The 1200 UTC RAMS forecast cycle predicted daily thunderstorm occurrence much better than 
the 0000 UTC cycle. 

• Thunderstorms were under-predicted in all gnd-4 zones of the 0000 UTC cycle, and in the 
southeastern zone of the 1200 UTC cycle. 

• Among the correctly predicted thunderstorm days, RAMS initiated the first thunderstorm correctly 
in one or more grid-4 zones about 50% of the time. Meanwhile, RAMS predicted the first daily 
thunderstorm to within 3 hours of actual initiation about 75% of the time. 

The AMU performed a variety of sensitivity tests to isolate the cause(s) of the RAMS objective error 
statistics, in particular the surface-based cold temperature bias. The only experiment that improved the cold bias 
(by about 3°C) involved running RAMS with an alternative radiation scheme that ignores the effects of clouds 
on incoming short-wave radiation. As a result of this experiment, the AMU found that RAMS generated 
widespread fog at the surface at all times over the ocean, and during the nocturnal hours over land. The fog 
could be the cause of the low-level cold bias since the fog reduces solar heating during the morning hours when 
the cold bias rapidly developed. The AMU has not identified the cause of this low-level fog in RAMS. 
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Introduction 


The Eastern Range Dispersion Assessment System (ERDAS) was developed by Mission Research 
Corporation (MRC)/ASTER Division (formerly ASTeR, Inc.) for the United States Air Force (USAF). ERDAS 
is designed to provide emergency response guidance for operations at the Kennedy Space Center (KSC) and 
Cape Canaveral Air Force Station (CCAFS) in the event of a hazardous material release or an aborted vehicle 
launch. ERDAS was delivered to the Eastern Range at CCAFS in March 1994. Under Applied Meteorology 
Unit (AMU) option-hours funding from the USAF Space and Missile Systems Center (SMC), ENSCO was 
tasked to evaluate the prototype ERDAS during the period March 1994 to December 1995. The evaluation 
report concluded that ERDAS provided significant improvement over current toxic dispersion modeling 
capabilities but contained a number of deficiencies. These deficiencies were corrected in the next generation of 
ERDAS that is part of the newly upgraded Meteorological and Range Safety Support (MARSS) replacement 
system. 

1.1 Task Background 

The MARSS replacement system contains an upgraded version of the Regional Atmospheric Modeling 
System (RAMS) that is designed to run on workstations with multiple processors. Developed at Colorado State 
University, RAMS is a dynamical numerical weather prediction model with optional parameterization schemes 
for representing physical processes in the atmosphere. The model may be run in two or three dimensions and in 
hydrostatic or non-hydrostatic modes. RAMS includes a terrain-following vertical coordinate, a variety of 
lateral and upper boundary conditions, and capabilities for mixed-phase microphysics. Details on the history, 
overview, and applications of RAMS can be found in Pielke et ai (1992) whereas a description of ERDAS can 
be found in Lyons and Tremback (1994). 

There are two main differences between the original and upgraded versions of the RAMS configuration in 
ERDAS. First, the original configuration of RAMS ran without cloud microphysics whereas the new 
configuration is run with full cloud microphysics on all grids. Second, the areal extent of the innermost, nested 
grid was expanded and the horizontal resolution was improved from 3 to 1.25 km. While the previous 
configuration of ERDAS was validated (Evans 1996), a systematic evaluation of the new configuration of 
ERDAS has not yet been performed. For this reason, representatives from 45th Range Safety (45 SW/SE) and 
45 th Weather Squadron (45 WS) requested that the upgraded version of RAMS in ERDAS be evaluated. 

The prognostic gridded data from RAMS is available to ERDAS for display and input to the Hybrid Particle 
and Concentration Transport (HYP ACT) model. The HYP ACT model provides three-dimensional dispersion 
predictions using RAMS forecast grids to represent the environmental conditions. Thus, the accuracy of 
dispersion predictions using the HYP ACT model is highly dependent upon the accuracy of RAMS forecasts. As 
a result, the primary goal of this evaluation is to determine the accuracy of RAMS forecasts during all seasons 
and under various weather regimes. 

The evaluation protocol is based on the operational needs of 45 SW/SE and 45 WS and designed to provide 
specific information about the capabilities, limitations, and daily use of ERDAS RAMS for operations at 
KSC/CCAFS. The ERDAS RAMS evaluation primarily concentrates on wind and temperature (stability) 
forecasts that are required for dispersion predictions using the HYP ACT model. The RAMS evaluation is 
divided into two segments, an objective and subjective component. The objective component focuses on model 
point error statistics at a number of observational locations. Since point error statistics cannot adequately 
evaluate meteorological phenomena and mesoscale patterns such as sea breezes and precipitation, there is also a 
subjective portion of the evaluation. The subjective component involves the manual examination of forecasts 
and observations to determine how RAMS predicts fine-scale phenomena such as sea breezes, precipitation, and 
thunderstorms. 

This report provides a summary of the AMU’s evaluation of the RAMS component of ERDAS for the 
1999_2000 cool season, and 1999 and 2000 warm seasons, focusing on local results at KSC/CCAFS and the 
immediate surrounding area. This report continues the work from the ERDAS RAMS interim report, which 
presented evaluation results from the 1999 Florida warm season (Case 2000). Therefore, this report will focus 
primarily on the 1999-2000 cool- and 2000 warm-season results. 
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Table 1.1. A summary of the grid parameters for all four RAMS grids. The model parameters include 
the number of grid points in the x, y, and z directions (nx, ny, and nz), horizontal resolution (dx), minimum 
and maximum vertical resolutions (dzmin and dzmax), and time steps (dt). 

Grid 

nx 

ny 

nz 

dx (km) 

dzmin (m) 

dzmax (m) 

dt (s) 

1 ] 

36 

40 

33 

60 

50 

■ 

45 

2 

38 

46 

33 

15 

50 


45 

3 

41 

50 

36 

5 

25 


22.5 

4 

74 

90 

36 

1.25 

25 


7.5 


1.3 RAMS Forecast Cycle 

RAMS is initialized twice-daily at 0000 and 1200 UTC using the Eta 12-h forecast grids and operationally- 
available observational data including the CCAFS rawinsonde (XMR), Aviation Routine Weather Reports 
(METAR), buoys, and KSC/CCAFS wind-tower, and 915-MHz and 50-MHz Doppler Radar Wind Profiler 
(DRWP) data. No variational data assimilation or nudging technique is applied when incorporating 
observational data. Instead, RAMS is initialized from a cold start by integrating the model forward in time from 
a gridded field without any balancing or data assimilation steps. Observational data are analyzed onto hybrid 
coordinates using the RAMS Isentropic Analysis (ISAN) package (Tremback 1990). The ISAN hybrid 
coordinate consists of a combination of constant potential temperature (isentropes) and terrain-following 
surfaces on which data are analyzed within the RAMS model domain, similar to the NCEP Rapid Update Cycle 
(RUC) model (Benjamin et al. 1998). 

The ERDAS RAMS forecast cycle is illustrated in Figure 1.2. The RAMS cycle is run in real-time for a 24- 
h forecast period on a Hewlett Packard (HP) K460 workstation cluster with 12 processors (with 1 of these 
processors serving as the master node). The model run-time performance is optimized by using a message- 
passing interface (MPI) on the 12 processors. In MPI, the run-time is significantly reduced compared to a single 
processor because each processor simultaneously performs computations on a portion of the domain (Tremback 
etal 1998). 

The operational cycle requires approximately 15 minutes to analyze observational data for the initial 
conditions using ISAN and 10-12 h to complete the 24-h forecast cycle. On many occasions when the model 
produced extensive convection (primarily during the summer months), a 24-h forecast could not complete in 
12 h due to the calculations associated with the microphysics scheme. In these instances, the existing RAMS run 
is terminated before the 24-h simulation is completed, and the new simulation begins. Consequently, RAMS 
data are occasionally missing from the 22-24-h forecasts, primarily due to extensive model convection. In the 
event of a 1 -cycle failure, prognostic data are still available from the previous forecast cycle. 

RAMS forecast output is available once per hour for display and analysis purposes due to disk-space 
limitations of the operational hardware. Thus, all portions of this model verification study are limited in time to 
a frequency of 1 hour, regardless of the frequency of available observational data. This frequency of model 
output presents a limiting factor in the verification since warm-season weather phenomena in Florida can 
develop over time scales much shorter than 1 hour (particularly convection). Nonetheless, hourly forecast 
output at high spatial resolution has the potential to provide valuable guidance in forecasting warm-season 
phenomena m east-central Florida. 
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2. Methodology 

The AMU evaluation of RAMS during the 1999-2000 cool, and 1999 and 2000 warm seasons includes 
both an objective and subjective component, following the methodology used in the interim ERDAS RAMS 
report. The objective component is designed to present a representative set of model errors of winds, 
temperature, and moisture for both the surface and upper-levels. The goal of the subjective verification is to 
provide an assessment of the forecast timing and propagation of the east-central Florida East Coast Sea Breeze 
(ECSB), daytime forecast precipitation, and forecast thunderstorm initiation by examining selected RAMS 
forecast fields. Since the 1999 warm-season objective and subjective results were thoroughly discussed in the 
interim report, this final report will focus on results from the 1999-2000 cool and 2000 warm seasons. 

2.1 Objective Component 

The objective component of the RAMS evaluation consists of five separate segments listed below that 
compute point error statistics: 

• Verification of the operational 4-grid configuration of RAMS. 

• Surface wind regime classification. 

• Thunderstorm day regime classification. 

• Comparison of point error statistics between the operational configuration and a RAMS 
configuration with a coarser horizontal resolution 

• Comparison of RAMS errors to the Eta model errors. 

Each portion of the objective component focuses on point error statistics at many different observational 
locations on all four forecast grids with emphasis placed on stations in grid 4. 


2.1.1 Standard Evaluation 


The standard objective evaluation consists of point forecast error statistics for the operational RAMS 
configuration during the 1999-2000 cool and 2000 warm seasons in Florida. Zero to 24-h point forecasts of 
wind, temperature, and moisture were compared with surface METAR and buoy stations, the XMR rawinsonde, 
KSC/CCAFS wind-tower, 915-MHz, and 50-MHz DRWP data at all available observational locations on grid 4, 
and selected surface and rawinsonde stations on grids 1—3. This report will focus on point error statistics at 
sensors within RAMS grid 4, particularly the KSC/CCAFS wind tower network, the XMR rawinsonde and the 
50-MHz profiler. The locations of all the observations used for point verification are given in Figure 2.1. 

The point statistics presented include the root mean square (RMS) error, bias, and error standard deviation 
(SD) of wind direction, wind speed, temperature, and dew point temperature for November 1999-March 2000 
(cool-season months), and for May-September 2000 (warm season months). In addition, the average values of 
forecasts and observations for these variables were computed as a function of forecast hour at all observational 
sites for the entire evaluation period. Special care was exercised when computing the mean and SD of wind 
direction errors following Turner (1986). However, in general, the mean seasonal observed and forecast wind 
direction quantities have little meaning because the distributions of the wind direction were nearly uniform, 
particularly during the warm season. Therefore, only plots of RMS error and bias are provided for wind 
direction. Error statistics for all other variables were calculated in the manner as outlined below. 


If O represents any forecast variable, then forecast error is defined as O' = O 0 , where the subscripts f 

and o denote forecast and observed quantities, respectively. The bias represents the average model error and is 
computed as 


Bias = 



( 2 . 1 ) 
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For purposes of interpretation in this report, we have assumed that the magnitude of the observational error 
is negligible compared to the model error. The total model error (RMS error) includes contributions from both 
systematic and random errors. Systematic error (bias) can be caused by a consistent misrepresentation of 
physical parameters such as radiation or model-generated convection. Random or nonsystematic errors, given 
by the error SD, represent the errors caused by uncertainties in the model initial condition or unresolveable 
differences in scales between the forecasts and observations. Note that the error standard deviation contains a 
component of natural variability since the model value is an average over a grid volume whereas the observed 
value is usually a point measurement. 

A quality control (QC) check was performed on all point error statistics to remove any errors greater than 3 
standard deviations from the mean error (bias). This QC check was performed to screen out bad observations or 
corrupted model forecasts and generally resulted in the rejection of less than 2 percent of all possible errors. 

2.1.2 Regime Classifications 

The second and third segments of the objective evaluation involve the computation of point error statistics 
under various weather regimes for the operational RAMS configuration during the 2000 warm season. 
Specifically, two types of regimes are examined in this report, surface winds and thunderstorm days. During 
each day, the surface wind regime was identified according to the early morning wind flow observed across the 
KSC/CCAFS wind-tower network. The days were then grouped into three classes of wind-flow patterns: 
westerly (offshore), easterly (onshore), and light or light and variable, where light wind regimes were defined as 
sustained speeds less than 5 knots. The RAMS forecasts were grouped together according to the similar surface 
wind regimes and error statistics were compiled for the similar wind regimes. 

For the thunderstorm regime classification, the RAMS point forecasts were computed for all combinations 
of forecast and observed thunderstorms on grid 4. Every day was categorized according to the occurrence of 
observed or forecast thunderstorms within the area of the RAMS grid 4. Each RAMS forecast fell into one of 
four categories: both observed and forecast thunderstorms, observed but no forecast thunderstorms, forecast but 
no observed thunderstorms, and no observed or forecast thunderstorms. Point error statistics were computed for 
each of these four combinations of thunderstorm days during the 2000 warm season. The results of the regime 
classification experiments are presented in Section 5.2.3. 

2. 1 .3 Benchmark Experiments 

The fourth and fifth segments of the objective evaluation consist of two benchmark experiments. One 
benchmark experiment compares the RAMS 4-grid operational configuration to a 3-grid configuration by simply 
excluding grid 4 and rerunning RAMS with only grids 1, 2, and 3. The statistics were computed separately for 
the 3-grid and 4-grid data for the forecast cycles when both models runs were available (refer to Appendix A). 
Section 4.5 presents the results of this comparison for observational data within the grid 4 domain. The goal of 
this experiment is to measure the impact from a decrease in horizontal resolution of the innermost grid on the 
resulting forecast errors. 

The second benchmark expenment compares the Eta point forecasts at 14 surface stations in the 
southeastern United States and 4 upper-air stations in the Florida peninsula to RAMS forecasts interpolated to 
the same stations (refer to Fig. 2.1 and Table 2.1). This important benchmark compares RAMS forecasts to the 
widely-used NCEP Eta model and quantifies any added value that may be provided by the RAMS over the Eta 
model, based on an objective comparison. This report will focus on the RAMS/Eta comparison at TTS (surface 
only), since this station is the only available location on RAMS grid 4. The results from the Eta model 
benchmark are discussed in Section 4.6. 
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traces (meteograms) of hourly temperature, dew point temperature, wind direction, wind speed, and mean sea- 
level pressure observations and RAMS forecasts were examined at seven selected surface stations on the east 
coast of the Florida peninsula (Jacksonville (JAX), Daytona Beach (DAB), the Shuttle Landing Facility (TTS), 
Melbourne (MLB), Vero Beach (VRB), West Palm Beach (PBI), and Miami (MIA)). Frontal passages were 
verified for both the 0000 UTC and 1200 UTC RAMS forecast cycles whenever the 24-h forecast overlapped an 
observed frontal passage and sufficient archived forecast and observed data were available. The discontinuities 
in winds, temperature, and dew point temperature were each verified independently because the wind shifts and 
temperature/dew point temperature gradients often occurred at different times with a frontal passage. 

The forecast errors associated with frontal passages were examined for three criteria, pre-frontal conditions, 
the frontal transition zone, and post-frontal maximum wind speed. The pre-frontal forecast wind direction, 
temperature, and dew point temperature errors were obtained by differencing the observations from the RAMS 
forecast values at each surface station that experienced both a forecast and observed frontal passage. The errors 
in forecast frontal timing were computed to the nearest hour, limited by the frequency of available RAMS 
output. The intensity of the frontal zone was verified by comparing the observed and forecast 3-h changes in 
each meteorological quantity following the initial observed or forecast discontinuity. Finally, the maximum 
forecast post-frontal wind speed was verified against the maximum observed post-frontal wind speed for each 
forecast that experienced a frontal passage. The results of the frontal verification are presented in Section 5.1.1. 

2. 2. L 2 Verification of Precipitation 

The 1999-2000 cool season offered only a small number of cases for frontal-associated precipitation 
verification. Very few significant frontal passages occurred with substantial frontal or pre-frontal rain bands, 
thereby preventing a robust verification of RAMS forecast precipitation during the cool season. The two most 
significant frontal events with precipitation across the Florida peninsula occurred on 2 November 1999 and 24 
January 2000, but each case had problems in the RAMS forecast. The 0000 UTC RAMS forecast cycle failed 
on 2 November. On the 24 January case, the NCEP Eta model poorly predicted the development of an Atlantic 
coastal cyclone prediction, and thus did not provide adequate boundary conditions for the RAMS forecasts. 
Additional cases with weak or broken squall lines occurred on a few days in March. 

Section 5.1.2 summarizes the most significant precipitation events for which sufficient forecast and 
observational data were available for subjective verification. In addition, a few RAMS forecasts are compared 
to observed satellite and rain-gauge data for the 24 January and 1 1 March 2000 events. To compare rain-gauge 
data to RAMS forecast precipitation, archived rainfall data from the St. John River, south Florida, and KSC 
networks were obtained from the Goddard Space Flight Center. An objective analysis was performed on the 
accumulated hourly rain-gauge data at every hour for the entire 1999-2000 cool season to obtain observed 
rainfall on RAMS grids 2, 3, and 4. RAMS forecast rainfall was then compared to these gridded rain-gauge data 
overlaid on infrared satellite imagery, primarily focusing on gnd-2 for peninsula-scale rainfall verification 
associated with fronts and troughs. 

2.2. 1.3 Verification of Low-level Temperature Inversions 

For all five cool-season months, the occurrence of observed and forecast temperature inversions at XMR 
were recorded in the lowest 3 km of the atmosphere, including both surface-based radiational inversions and 
elevated subsidence inversions. Approximately 80% of the cases examined were surface-based radiational 
inversions. Both the 0000 UTC and 1200 UTC RAMS forecast cycles were examined and verified against the 
observed morning XMR sounding, which was typically released at 1115 UTC. Therefore, the 11-h forecast 
from the 0000 UTC RAMS cycle and the 23-h forecast from the previous day’s 1200 UTC RAMS cycle are the 
verifying forecast hours. 

The number of model forecast ‘‘‘hits 1 ’ and “misses” were compiled to determine how well RAMS can predict 
the occurrence of a temperature inversion. When both an observed and forecast inversion occurred, specific 
parameters were verified including the intensity of the temperature inversion in °C, the height of the inversion 
base in meters, and the depth of the inversion layer in meters. 
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In addition to the evaluation of the RAMS predicted ECSB at the 12 selected towers, the AMU conducted 
two benchmark/sensitivity tests within the sea-breeze verification. The first sensitivity study compares the 
RAMS 4-grid sea-breeze forecasts to RAMS 3-grid forecasts, where grid 4 is simply excluded in the model 
predictions. The sea-breeze verification is conducted at the 12 wind towers (Figure 2.2) for all common 4-grid 
and 3-grid RAMS forecasts only during the 2000 warm season (refer to Appendix A for available RAMS 
forecasts). The second sensitivity experiment compares the RAMS 4-grid sea-breeze forecasts to the Eta point 
forecasts at the Shuttle Landing Facility (TTS, shown in Fig. 2. Id). The TTS sea-breeze verification was 
performed for all common RAMS and Eta point forecasts during the 1999 and 2000 warm seasons. The same 
onshore versus offshore criterion as used for the KSC/CCAFS wind-tower evaluation was applied when 
verifying the RAMS and Eta sea-breeze forecasts at TTS. 



Figure 2.2. A plot of the geographic locations of KSC, CCAFS, the local water bodies surrounding 
KSC/CCAFS, and the 12 KSC/CCAFS wind towers used for the east coast sea-breeze subjective verification. 
The labels of geographical locations are given in a) and the wind tower locations are shown in b). The wind 
towers were chosen to examine the timing of the ECSB along the immediate Atlantic coastline, Merritt Island, 
and mainland Florida. 

A 2 x 2 contingency table was used to summarize the ECSB verification statistics based on the occurrence 
of both an observed and forecast ECSB at any of the 12 KSC/CCAFS towers. A “hit” is defined as the 
occurrence of both an observed and forecast sea-breeze passage at a particular KSC/CCAFS tower. Because 
RAMS forecast output is available once per hour, the AMU verified the timing of the onset and movement of the 
sea breeze to the nearest hour at each of the 12 KSC/CCAFS towers. Table 2.2 is a sample 2x2 contingency 
table from which a variety of categorical and skill scores can be computed to measure forecast performance. 
The total number of correct forecasts is given by x in the upper left comer (forecast and observed = yes) and w 
in the lower right comer (forecast and observed = no). The number of forecast failures is given in the lower left 
portion of the table (forecast = no, observed = yes) and the number of false alarm forecasts is given in the upper 
right comer (forecast = yes, observed = no). 


Table 2.2. A sample 2x2 contingency table for the evaluation of a 
forecast element is shown from which categorical and skill scores are computed 
(see text). 


Observed — Yes 

Observed = No 

Forecast — Yes 

X 

z 

Forecast = No 

y 

w 

Number of correct forecasts = (x+w) 
Number of false alarm forecasts = z 
Number of forecast misses = y 
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Because convective precipitation is often localized and fleeting across east-central Florida, the AMU 
applied an alternative technique to verify precipitation on RAMS grid 4 during the 2000 Florida warm season. 
Based on the methodology described in Manobianco and Nutter (1999), the RAMS grid 4 was divided into six 
separate zones to identify locations of forecast and observed precipitation (Fig. 2.3). The original intent was to 
verify hourly accumulated forecast precipitation; however, the AMU discovered late in the study that 
instantaneous precipitation rates were being archived once per hour rather than RAMS accumulated rainfall. As 
a result, only instantaneous precipitation rates were verified against instantaneous observed precipitation once 
per hour. Consequently, this precipitation verification will be quite stringent compared to a technique that 
utilizes accumulated observed and forecast precipitation fields. The effects of this instantaneous rainfall 
verification compared to a verification of accumulated precipitation were not estimated. 

In order to perform the precipitation verification adequately, the AMU could rerun RAMS forecasts from 
May to September 2000 and extract the accumulated precipitation fields from the model. In addition, the AMU 
could obtain archived WSR-88D rainfall estimates from the MLB radar site for the entire warm season to serve 
as a validation for the predicted precipitation. However, the level of effort required to perform this analysis is 
beyond the available resources of the current task. 

Hourly forecast precipitation rates > 5 mm If 1 (0.2 in h' 1 ) were identified on a daily basis for both the 0000 
and 1200 UTC RAMS runs in each verification zone between 1500-2300 UTC (peak convective hours). The 
technique used composite reflectivity at 2,000 ft from the WSR-74C to verify the location of observed 
precipitation at each hour. A “hit” was defined as the occurrence of both forecast and observed precipitation 
within a specific zone at a given hour, with intensity > 5 mm h V To obtain an approximate reflectivity threshold 
corresponding to a 5 mm hr 1 rain rate, the AMU utilized the tropical reflectivity/rain rate relationship derived in 
the Global Atmospheric Research Program (GARP) Atlantic Tropical Experiment (GATE, Hudlow 1979), 

Z = 230/? 1 25 , (2.9a) 

dBZ = 101og 10 230 + 12.51og 10 R , (2.9b) 

where Z is reflectivity and R is rain rate in mm h 1 . Using this relationship yields a reflectivity factor of about 32 
dBZ corresponding to a rain rate of 5 mm h' 1 . 

The AMU also verified precipitation in two and three-hourly verification bins in order to identify any 
improvement in precipitation forecast skill with a larger verification time interval. Increasing the verification 
time interval and reapplying skill-score thresholds can help to determine a predictable limit of RAMS 
precipitation forecasts. For the two-hour forecast bins, a hit is defined as the occurrence of observed and 
forecast precipitation in the same zone for either hour of the two-hour bin. Similarly for three-hourly bins, a hit 
is defined as the occurrence of both observed and forecast precipitation at any hour within the three-hour time 
interval. To measure the accuracy of RAMS hourly precipitation forecasts, the AMU computed the POD, FAR, 
CSI, and HSS for each zone in Figure 2.3. 

In this study, precipitation is verified according to the occurrence of precipitation at only 1 threshold (> 5 
mm hr *) anywhere within the six separate zones of Figure 2.3. Thus, the methodology used in this study is less 
stringent than threat score techniques in terms of spatial and intensity verification. However, this methodology 
is more stringent temporally based on the small time windows used to verify the forecast precipitation (1—3 h). 
In addition, this study looks at instantaneous precipitation rates rather than accumulated precipitation. Most 
current operational techniques still verify model precipitation for 24-h periods; however, NCEP recently began 
routine 3-h precipitation verification for many national-scale operational models (Baldwin 2000). Section 5.2.2 
summarizes the categorical and skill scores of RAMS forecast precipitation and includes verification scores for 
1-, 2-, and 3-h verification windows. 
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correctly predicted the initiation time exactly (0-h difference between observations and forecast), within 1 h (-1 
to +1 h error), within 2 h (-2 to +2 h error), and within 3 h (-3 to +3 h error) of the observed time, irrespective of 
spatial accuracy. In addition, the spatial and timing accuracy of thunderstorm initiation were examined in 
combination by developing contingency tables and determining categorical and skill scores for each individual 
grid-4 zone based on specific timing thresholds. The results of the thunderstorm initiation verification are 
presented in Section 5.2.3. 
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DRWP was out of service or unreliable for a significant portion of the season due to damage inflicted by two 
hurricanes during the fall of 1999 (Floyd and Irene). As a result, less than half the possible verification points 
were available for the 1999-2000 cool season (Fig. 3.2a), and these data will not be used for upper-air wind 
verification in the cool season. However, the data were much more reliable during the 2000 warm season, with 
> 70% of the maximum verification points available in much of the mid-upper troposphere (Fig. 3.2b). 



Figure 3.2. Time-height plot of the percentage of available 50-MHz DRWP verification data for the 1 200 
UTC forecast cycle during the a) 1999-2000 cool season (November— March) and b) 2000 warm season 
(May-September). 


The XMR rawinsonde standard launch times are typically 0615 and 1815 local time year-round, with an 
additional sounding launch at 1100 local during the convective season (May-September). Thus, XMR 
rawinsonde observations were available at 1115 and 2315 UTC during the 1999-2000 cool season, and 1015, 
1500, and 2215 UTC during the warm season. Since RAMS forecast output was available only once per hour, 
the observations were used for verification at both hours surrounding the observation, when appropriate. 

The distribution of available XMR verification data for the 1200 UTC RAMS forecast cycle is shown in 
Figures 3.3 and 3.4 for wind direction and temperature, respectively. During the cool season, the 1200 UTC 
RAMS predictions interpolated to the XMR site were verified for forecast hours 0, 11, 12, 23, and 24 
(corresponding to universal times 1200, 2300, 0000, 1100, and 1200 UTC). The theoretical maximum number 
of verification points at a particular time and sounding level is equal to the number of possible verification days, 
152 in the cool season and 153 in the warm season. However, the largest number of verification points is about 
130 in conjunction with the late-moming sounding launches during 2000 warm season. A combination of 
missing sounding data and failed RAMS forecasts (see next section and Appendix A) account for the loss of 
data during the two seasons. 

The available wind verification data extends from the surface to — 1 0 — 1 2 km (Fig 3.3) whereas the 
temperature verification data extend from the surface to about 8 km (Fig. 3.4). Available dew point temperature 
verification data only reached ~6— 7 km (not shown). The cause of this lack of data above certain levels for each 
variable is not known. A bug may exist in the RAMS data extraction routine since sounding data are readily 
available above these levels. 
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The operational RAMS configuration is designed to provide each forecast cycle with 12 hours to complete a 
24-h forecast (see Section 1.3). Due to limitations in the processing speed associated with the operational 
hardware, the current configuration of RAMS cannot always complete all 24 forecast hours within the 
prescribed time of 12 hours. If the RAMS prediction takes longer than 12 hours to complete, the current cycle is 
terminated prematurely, and the subsequent cycle begins. Thus, a recurring issue with the operational RAMS 
configuration is the termination of the current forecast cycle before it completes all 24 forecast hours. This loss 
of data was most prevalent during the warm seasons when extensive convection generated by the model slowed 
down the system due to the intensive computations associated with the microphysical cloud scheme. 

Figure 3.5 illustrates the availability of RAMS data as a function of forecast hour for both the 1999-2000 
cool season and 2000 warm season. Since widespread precipitation only occurred rarely during the 1999-2000 
cool season, most RAMS forecasts completed all 24 hours of integration as only about 10—25 forecasts from 
each cycle failed to complete the 24th hour (Fig. 3.5a). The 2000 warm season statistics depict quite a different 
story since well over half of all successful forecasts did not complete the 24th forecast hour (Fig. 3.5b). In fact, 
a substantial drop in available data begins at 22 hours and continues to drop sharply out to 24 hours. The 
variations of RAMS data availability between the cool and warm seasons clearly indicate the slower RAMS 
performance during the active convective months. 



Figure 3.5. A display of the number of available RAMS forecasts as a function of forecast hour for the a) 
1999-2000 Florida cool season (November-March) and b) 2000 Florida warm season (May-September). The 
gray bars represent the number of days of available 0000 UTC cycle forecasts whereas the black bars are the 
number of days of available 1200 UTC cycle forecasts. The theoretical maximum numbers at each hour are 152 
for the 1999-2000 cool season and 153 for the 2000 warm season. 
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4.1.2 Summary of Upper-level Errors (XMR rawlnsonde) 


Table 4.2. Summary of the 4-grid RAMS upper-air error statistics for the 0000 and 1200 UTC forecast 
cycles during the 1999-2000 Florida cool season. Upper-level conditions are verified at the XMR rawinsonde 
for the 1 1 15 UTC and 2315 UTC standard launch times (1 1-h and 23-h forecasts, respectively). 

Variable 

Forecast Cycle 

RMS Error 

Bias 

Notable Errors 





• 

Too cold below 650 mb in both cycles. 





9 

During the early morning hours, RAMS is 


0000 UTC 

1 to 3.5 

-3 to 1 


too warm at the surface and much too cold 

Temp 

(0 C \ 





just above the surface, indicating that the 


1200 UTC 

1 to 4 

-3 to 1 


lowest levels are not stable enough. 





9 

Slightly too warm above 650 mb. 





9 

Largest RMS error near the surface. 





9 

Too moist at low levels below 900 mb. 

Dew 

0000 UTC 

4 to 8 

-4 to 3 

9 

Largest RMS error near 850 mb. 

Point 




9 

Tendency to be too dry at mid and upper 

(°C) 

1200 UTC 

3 to 8 

-4 to 2.5 


levels. 





9 

Largest errors below 950 mb. 





9 

Smallest errors between 300^400 mb. 


0000 UTC 

15 to 60 

-20 to 20 

9 

Positive bias near the surface; negative bias 

Wind 





above 950 mb. 

Direction 
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In the 0000 UTC cycle, an easterly bias 

(deg.) 

1200 UTC 

15 to 55 

-15 to 25 


(negative u-wind bias) increases with time at 






all levels. 





• 

In both cycles, a southerly bias (positive v- 






wind bias) occurs at both times and all levels. 





• 

Smallest errors near the surface, largest at 

Wind 

0000 UTC 

2.5 to 4.5 

-2 to 1.5 


upper levels. 

Speed 




• 

Positive bias near the surface, negative bias 

(m s' 1 ) 

1200 UTC 

2.5 to 5.5 

-2 to 2 


above 950 mb. 


4.1.3 Discussion of 1999-2000 Cool-season Results 

Because the diurnal variations in objective error statistics were quite similar for the operational 0000 and 
1200 UTC forecast cycles, a brief discussion of the RAMS point forecasts is provided here for only the 1200 
UTC forecast cycle surface and upper-level error statistics on RAMS grid 4. The analysis of surface errors 
focuses on the results from the KSC/CCAFS wind towers, but also supplements these results with standard 
METAR observational errors from grid 4 and the errors at the two buoys of Cape Canaveral, FL (refer to Fig. 
2.1 for station locations). All referenced graphical plots and figures in this section can be found in Appendix B. 

4.1. 3.1 Surface errors from 1200 UTC Cycle (KSC/CCAFS towers, METAR, buoys) 
a. Temperature and dew point temperature 

During the 1999-2000 Florida cool season, a pronounced daytime cold temperature bias was prevalent at all 
verification sensors of interest, including the KSC/CCAFS wind-tower network, METAR observations on grid 
4, and the two offshore buoys. At the KSC/CCAFS wind towers, a slight warm bias (~2°C) occurs in the initial 
condition, but quickly switches to a cold bias by the 2-h forecast, reaching a maximum magnitude of -4°C at 6 h 
(Fig. Blc). The bias dissipates to 0°C during the overnight horns after 18 h. The combination of the random 
error (SD) and the bias yields a maximum RMS error close to 5°C at the 6-h forecast (Fig. Bib). As shown in 
Figure B2, the surface cold bias is also quite significant, as well as constant throughout all 24 forecast hours at 
the offshore buoys. The mean forecast temperature at the buoys is about 3-4° too cold compared to the 
observed temperatures yielding a bias of the same magnitude (Fig. B2a). The combination of the cold bias and 
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a. Temperature and dewpoint temperature 

RAMS generally produced a forecast temperature profile that is too stable throughout the troposphere for 
both XMR launch times. At 2315 UTC (1 1-h forecast), a cold bias occurs from the surface to about 625 mb 
whereas a slight warm bias is found above 625 mb (Fig. B6a). In the early morning forecast (23 h compared 
with the 1 1 15 UTC sounding), RAMS still has a temperature profile that is too stable from about 950 mb to 625 
mb; however, at the very lowest levels (surface to 950 mb), the bias profile shows that RAMS is not stable 
enough (Fig. B6b). A slight warm bias exists at the surface while at 950 mb the temperatures are too cold by 
nearly 3°C. This result suggests that on average, RAMS does not adequately predict the occurrence and/or 
intensity of nocturnal and early morning surface-based temperature inversions during the 1999-2000 cool 
season. This conclusion implied by the low-level bias profile is supported by the subjective verification of 
RAMS low-level temperature inversions, presented in Section 5.1.3. 

In the dew point temperature forecasts, RAMS had the largest RMS errors at mid-levels (600—850 mb), the 
largest moist bias near the surface, and the largest dry bias in the 450-600 mb layer (1 1-h forecast shown in Fig. 
B7). The 23-h forecast exhibited similar characteristics but with a larger moist bias near the surface and a larger 
dry bias above 600 mb (not shown). 

b. Winds . 

As a rule of thumb, the wind direction RMS errors decrease with height whereas the wind speed RMS errors 
increase with height. The 1 1-h RAMS forecast wind direction RMS error in Figure B8a shows a 50° RMS error 
near the surface, decreasing to about 10° near 300 mb. Meanwhile, a positive bias occurs at the surface with a 
general small negative bias above 950 mb (Fig. B8b). Consist with previous studies (Manobianco et al 1996; 
Nutter and Manobianco 1999; Case 2000), the wind speed errors increase as wind speed increases. At 1 1 h 
(2300 UTC), the strongest mean wind speeds occur above 400 mb (Fig. B9a) corresponding to the greatest 
magnitude of errors (Figs. B9b-d). A positive bias in wind speed is evident very close to the surface while a 
negative wind speed bias prevails above 950 mb (Fig. B9c). Similar error patterns occur in the 23-h RAMS 
wind forecasts as well (not shown). 

The individual wind component errors (not shown) indicate that RAMS consistently over-predicted the 
southerly (v-wind) component above 950 mb by 2 m s 1 or more for both the 1 1-h and 23-h forecasts. The 
predicted u-wind component experienced a negative (easterly) bias above 700 mb at 11 h, but exhibited no 
significant bias at 23 h (not shown). 

4.1.4 Low Temperature Verification 

As part of the 1999-2000 cool-season evaluation, the AMU was tasked to verify the RAMS forecast low 
temperatures on the innermost forecast grid. To accomplish this verification, the AMU developed an algorithm 
that identifies the lowest hourly forecast and observed surface temperature at the 1.8-m (6-ft) level for all 
available KSC/CCAFS wind towers. In addition, the lowest hourly observed and forecast temperature was 
identified at selected surface METAR stations, including Orlando (MCO), Daytona Beach (DAB), Melbourne 
(MLB), and the Shuttle Landing Facility (TTS). The subsequent lowest forecast hourly temperature was 
verified for every available station during all successful RAMS forecasts (both 0000 and 1200 UTC cycles). 
Only the results from the 0000 UTC forecast cycle are presented in this section. 

The low temperature algorithm works as follows. The daily point forecast and observed temperatures at 
individual wind towers or METAR stations were examined between the hours of 0000 and 1200 UTC. Prior to 
identifying the lowest hourly temperature, data availability and QC checks were established to ensure a 
sufficient amount of quality data. For data availability, over half of the hourly data in the 0000-1200 UTC 
range must be present, and at least half of these existing files must contain the station of interest. For QC 
checks, a realism test and a simple temporal buddy check were used to ensure that unrealistic data values were 
not used in the statistics. 

Table 4.3 summarizes the results of the low-temperature verification on RAMS grid 4 for the 0000 UTC 
RAMS cycle. The RMS errors for forecast low temperature range from 1.7°C at MLB to 2.5°C at DAB and the 
KSC/CCAFS wind towers. No significant bias occurs at any of the locations. The timing RMS errors fall 
between 3.0 h at MCO and 3.9 h at both DAB and MLB. The forecast low temperatures are typically reached 
too early at each of the METAR stations, indicated by the negative timing bias. However, the only timing biases 
comparable in magnitude to the frequency of available output (once per hour) occurred at DAB and MLB (-2.0 
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4.2.2 Summary of Upper-level Errors (XMR rawinsonde and 50-MHz profiler) 


Table 4.5. Summary of the 4-grid RAMS upper-air error statistics for the 0000 and 1200 UTC forecast 
cycles during the 2000 Florida warm season. Upper-level winds were verified at the KSC/CCAFS 50 MFlz 
profiler and XMR rawinsonde whereas the upper-level temperatures and dew point temperatures were verified 
at XMR only. 

Variable 

Forecast Cycle 

RMS Error 

Bias 

Notable Errors 

Temp 

(°C) 

0000 UTC 
1200 UTC 

1 to 4 
1 to 3 

-3.5 to 0.5 
-2.5 to 0.2 

• Early morning is not stable enough in the 
lowest 50 mb (as in cool season). 

• A general cold bias at all levels, but largest 
at low levels. 

• The magnitudes of errors decrease with 
height. 

Dew 

Point 

(°C) 

0000 UTC 
1200 UTC 

1.5 to 7 
1 to 5.5 

-2.5 to 1.5 
-2 to 0.5 

• The largest RMS errors occur in mid-levels 
(600-800 mb). 

• Tendency towards a negative (dry) bias at 
low- and upper-levels with comparable 
magnitudes. 

Wind 

Direction 

(deg.) 

0000 UTC 
1200 UTC 

20 to 60 
20 to 70 

-15 to 10 
-20 to 20 

• At 50-MHz profiler: the largest errors occur 
during the afternoon hours between 2-6 km. 

• At XMR: the largest errors are found near 
the surface during the early morning hours. 

• An easterly (negative u-wind) bias is found 
at low levels. 

• A general southerly (positive v-wind) bias 
occurs at all levels, especially above 12 km 
at the 50-MHz profiler. 

Wind 
Speed 
(m s' 1 ) 

0000 UTC 
1200 UTC 

1.5 to 7 
1.8 to 7 

-2 to 2 
-2 to 2 

• The largest errors occur at upper-levels 
where wind speeds are strongest. 

• At XMR: a positive bias occurs in the lowest 
100 mb. 

• A negative bias occurs above 850 mb. 


4.2.3 Discussion of 2000 Warm Season Results 

A brief discussion of the RAMS point forecasts is provided here for the 1200 UTC forecast cycle surface 
and upper-level error statistics on RAMS grid 4. Again, the analysis of surface errors focuses on the results 
from the KSC/CCAFS wind towers, but are also supplemented with standard METAR observational errors from 
grid 4 and the errors at the two buoys of Cape Canaveral, FL. All referenced graphical plots and figures in this 
section can be found in Appendix C. 

4.2.3. 1 Surface errors from 1200 UTC Cycle (KSC/CCAFS towers, METAR, buoys) 
a. Temperature and dew point temperature 

As in the 1999-2000 cool season, RAMS exhibits a surface-based cold temperature bias at all observational 
sensors on grid 4. The maximum magnitude of the cool bias during the 2000 warm season is about 1°C smaller 
than during the cool season, and predominantly a daytime phenomenon as well (particularly over land). Figures 
Cl and C2 summarize the evolution of the 1200 UTC surface temperature errors over the KSC/CCAFS wind 
towers and offshore buoys, respectively. The RMS error at the KSC/CCAFS wind towers peaks at the 8-h 
forecast (Fig. Clb) and the cold bias peaks between 8-1 1 h (Fig. Clc). At the offshore buoys, the RMS error 
and cold bias steadily increase from 1—9 h, then maintain a constant error thereafter (Fig. C2b,c). Interestingly, 
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the RMS error also increases from 650 mb down to the surface (Fig. C7b). The temperature errors are nearly 
constant above 650 mb in the 10-h forecast as well as the 3-h and 22-h forecasts (not shown). Except for the 
surface, the SD of the errors is nearly constant at about 1°C as a function of height for all three forecast times at 
about 1°C. Thus, the primary contributor to the growth of the total error at low levels is the model’s systematic 
cold error. 

The profile of dew point temperature errors during the 2000 warm season is also quite similar to the error 
profile during the 1999-2000 cool season. In all three forecast times verified at XMR, the dew point errors are 
smallest near the surface (1.5-3.0°C) and increase with height reaching a maximum of 5-6°C between 600-800 
mb. then decrease above 600 mb (not shown). The errors are composed of primarily random variability since 
the SD is quite similar to the RMS error and the biases are small in magnitude relative to the RMS error. 

b. Winds 

The errors in wind speed and wind direction exhibit a similar pattern to the 1999-2000 cool-season errors, 
but the change in magnitude of the errors as a function of height is not as pronounced as the cool season, 
particularly with wind direction. In the lowest 2 km (roughly surface to 800 mb) at XMR, the magnitude of the 
wind speed errors increase slightly with time from about 2 m s 1 at the 3-h forecast to 2.5 m s at the 10-h and 
22-h forecasts (not shown). The wind speed errors are nearly constant with height in the lowest 2 km as well. 

Above 2 km, the wind speeds can be verified against the 50-MHz DRWP on an hourly basis. Figure C8 
shows a time-height cross section of RAMS wind speed errors at the 50-MHz DRWP location. By comparing 
the mean observed and forecast wind speeds in Figures C8a and b, it appears that the RAMS wind speeds are 
slightly weaker than the observed speeds. The plot in Figure C8d reveals a 1—2 m s negative bias at nearly all 
levels particularly after the 12-h forecast. Meanwhile, the total error in Figure C8c depicts an increase in RMS 
error with height reaching a maximum between 12-14 km, corresponding to the axis of maximum observed 
wind speed. 

The wind direction errors verified at XMR in the lowest 2 km (roughly surface to 800 mb) indicate that 
RAMS forecasts have the largest errors near the surface for the 3-h and 22-h forecasts (1500 and 1000 UTC 
respectively shown in Figs. C9a and Cl la). At these times, the RMS error decreases with height from the 
surface to 800 mb. Meanwhile, the wind direction errors during the late afternoon, post-sea breeze regime have 
a much different profile between the surface and 800 mb (Fig CIO). During the mean post-sea breeze regime, 
the wind direction RMS errors increase from about 30° near the surface to about 50-55° between 800-900 mb. 
This late-aftemoon error structure could result from consist accurate predictions of the sea-breeze onset and 
propagation across east-central Florida near the surface (refer to the results of the east-coast sea-breeze 
subjective verification in Section 5.2.1), combined with uncertainty in the depth of the sea-breeze circulation. 

The wind direction verification above 2 km at the 50-MHz profiler is shown in Figure Cl 2. A general 
increase in the RMS error occurs in the first 6 forecast hours at nearly all vertical levels (Fig. Cl 2a). Most of 
the errors are between 30-50° for all levels and forecast hours at the 50-MHz DRWP. The magnitude of the 
bias is typically 10° or less (Fig. Cl 2b) and thus, substantially less than the magnitude of the RMS error. As a 
result, most of the errors are composed of random variability. 

The wind component errors do not have many extraordinary characteristics that are substantially different 
from the wind speed and direction errors. The RMS errors increase with height for both wind components (as in 
the wind speed) and only a 1 ms 1 easterly bias occurs below 950 mb in the 10-h forecast at XMR (not shown). 
The v-wind component tends to have the largest bias at all times and levels, especially in the lowest levels of the 
atmosphere. Generally, a 1—2 m s' 1 southerly bias is evident with the greatest bias occurring in the 10-h forecast 
below 800 mb at XMR (not shown). 
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Figure 4.1. A meteogram plot of the 1200 UTC RAMS temperature errors (°C) during westerly, easterly, 
and light surface wind regimes for the 2000 Florida warm season. The temperature is verified at the 1.8-m level 
of the KSC/CCAFS wind-tower network. Parameters plotted as a function of forecast hour are a) mean forecast 
temperature under each wind regime, b) RMS error, c) bias, and d) error standard deviation. 

Wind Direction 

The results of the wind-regime classification reveal two very apparent characteristics of the wind direction 
errors. First, the westerly wind regime contains the largest RMS error during the afternoon and evening hours, 
likely associated with the higher frequency of convection under low-level westerly flow. Second, the light wind 
regime is the primary contributor to the relatively large RMS errors during the late night and early morning 
hours, as anticipated. Figures 4.2 and 4.3 show the wind direction RMS errors and biases for the 0000 and 1200 
UTC forecast cycles, respectively. In both forecast cycles, the daytime errors are substantially larger associated 
with westerly wind flows compared to light or easterly winds. Under surface westerly wind flow, the 0000 UTC 


29 






1200 UTC Sfc. Wind Cycle Wind Dir (deg) for 16.5-m towers 



0 3 6 9 12 15 18 21 24 


Forecast Hour 


Figure 4.3. A meteogram plot of the 1200 UTC RAMS wind direction errors (deg.) during westerly (solid 
line), easterly (a), and light surface wind regimes (*) for the 2000 Florida warm season. The wind direction is 
verified at the 16.5-m level of the KSC/CCAFS wind-tower network. Parameters plotted as a function of 
forecast hour are a) RMS error, and b) bias. 

Wind Components 

The distribution of mean u-winds in the KSC/CCAFS wind tower network during the 0000 UTC cycle is 
shown in Figure 4.4a. By examining each of these mean wind flows, the mean forecast sea-breeze passage is 
evident at -1300 UTC under easterly flow, -1400 UTC under light flow, and -1600-1700 UTC under westerly 
flow. Under westerly wind flow, the mean u-wind is between 0—2 m s 1 from 2—16 h, then becomes negative 
with the mean passage of the sea breeze thereafter. In light regimes, the u-wind approaches zero at about 12 h, 
but generally maintains an easterly component at all hours. Meanwhile under easterly surface flow, the mean u- 
wind remains under -2ms 1 at all hours. RAMS predicts easterly flow following the mean sea-breeze passage 
with nearly the same intensity in both the light and easterly wind regimes (Fig. 4.4a). 

While no dramatic variations are evident in the v-wind errors under different wind regimes (not shown), the 
u-wind errors show some interesting behavior in both the 0000 and 1200 UTC forecast cycles. In the 0000 UTC 
cycle, the largest u-wind RMS errors occur after 15 h under westerly flow (Fig. 4.4b). The light wind regime 
has somewhat smaller errors while the easterly wind flow has the smallest errors of all three regimes during the 
afternoon and evening hours. All three regimes have comparable errors between 0-15 h. A similar pattern is 
evident in both the bias and SD plots in Figures 4.4c and d. A negative (easterly) bias occurs in all regimes, 
especially after 15 h, but is largest under westerly flow and smallest in easterly flow. Similarly, the SD 
(random) errors are by far the largest under westerly flow and smallest in easterly flow. The v-wind random 
errors are generally largest under westerly flow during the afternoon hours as well (not shown). 

This relatively large random u-wind error during westerly flow could be the result of two factors. First, 
when surface winds are sufficiently strong from the west, the ECSB typically remains close to the east coast of 
Florida within the KSC/CCAFS wind-tower domain. If the RAMS model has just a small error in the location 
or timing of the ECSB then large random errors in the u-wind can result in the wind-tower network. Second, as 
mentioned previously, convection is most prevalent in east-central Florida under westerly flow since the 
focusing mechanism for convection (i.e. the ECSB) remains near KSC/CCAFS. Errors between observed and 
model-predicted convection can also lead to large random wind errors. Similar features in the u-wind errors are 
also found in the 1200 UTC cycle (Fig. 4.5); however, the biases are nearly identical compared to the 0000 UTC 
forecasts. Again, the random u-wind errors are largest under westerly flow in the 1200 UTC cycle (Fig. 4.5d). 
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Figure 4.5. A meteogram plot of the 1200 UTC RAMS u-wind component errors (m s ') during westerly 
(solid line), easterly (a), and light surface wind regimes (*) for the 2000 Florida warm season. The u-wind is 
verified at the 16.5-m level of the KSC/CCAFS wind-tower network. Parameters plotted as a function of 
forecast hour are a) mean forecast u-winds under each wind regime, b) RMS error, c) bias, and d) error standard 
deviation. 

4.3.2 Thunderstorm Regime 

For the thunderstorm day regime classification, the RAMS forecasts were grouped together according to the 
observed versus forecast thunderstorm days using the subjective thunderstorm verification results (Section 
5.2.3). A contingency table of the daily thunderstorm occurrence during the 1500-2300 UTC time frame was 

developed for both the 0000 and 1200 UTC forecast cycles as part of the subjective evaluation scheme (Table 
4.7). The forecasts composing each quadrant of the 0000 and 1200 UTC contingency tables were grouped 
together and point forecast error statistics at the KSC/CCAFS wind towers were calculated separately for each 
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Figure 4.6. A meteogram plot of 1200 UTC RAMS temperature errors (°C) during the four contingency 
combinations of thunderstorm forecasts (yes-yes, yes-no, no-yes, no-no) for the 2000 Florida warm season. The 
temperatures are verified at the 1.8-m level of the KSC/CCAFS wind-tower network. Parameters plotted as a 
function of forecast hour are a) mean forecast temperatures, b) RMS error, c) bias, and d) error standard 
deviation (SD). 
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4.4 Comparison between 4-grid and 3-grid RAMS Configurations 

This section presents the results from a comparison between the 4-grid RAMS and a 3-grid configuration of 
RAMS, where grid 4 was withheld. This experiment was conducted to determine the impact of a reduction in 
horizontal resolution of the innermost grid on objective error statistics. The error statistics are compared at the 
KSC/CCAFS wind towers for the surface, and at XMR and the 50-MHz DRWP for upper-levels. Since the 50- 
MHz DWRP data were not of sufficient quality during the 1999-2000 cool season, these errors are not shown. 


4.4.1 1999-2000 Cool Season 

4.4AA Summary of 1200 UTC Surface Errors (KSC/CCAFS wind towers) 


Table 4.8. Summary of the 4-grid/3-grid RAMS error comparison at the KSC/CCAFS wind 
towers for the 1200 UTC forecast cycle during the 1999-2000 Florida cool season. Temperature and dew 
point errors are valid at 1 .8 m whereas wind direction and speed errors are valid at 16.5 m. 



RMS Error 

Bias 

Notable Errors 





• Similar errors at all times, but slightly larger 

Temp 

4-grid 

1.5 to 4.5 

-3.5 to 1 

in the 3-grid forecasts during the day. 

(°C) 

3-grid 

1.5 to 5 

-4 to 1 

• Both configurations have notable daytime 




cold biases. 





• A slightly larger daytime RMS error and dry 

Dew 

4-grid 

1.5 to 3.5 

-1.5 to 1 

bias occurs in the 3-grid forecasts. 

Point 




• The 4-grid has a slightly more moist bias 

(°C) 

3 -grid 

1.5 to 3.5 

-2 to 0.5 

during the nocturnal hours. 

Wind 

4-grid 

15 to 60 

-20 to 15 

• Errors are virtually identical at all forecast 

Direction 




hours. 

(deg.) 

3-grid 

15 to 60 

-20 to 20 


Wind 

4-grid 

1.2 to 2 

-0.5 to 0.8 

• A slightly higher positive bias occurs in the 3- 

Speed 




grid forecasts; otherwise, the errors are nearly 

(m s' 1 ) 

3 -grid 

1.2 to 2.3 

Oto 1.5 

the same. 


4,4 A. 2 Summary of 1200 UTC Upper-level Error (XMR rawinsonde) 


Table 4.9. Summary of the 4-grid/3-grid RAMS error comparison at the XMR rawinsonde site for 
the 1200 UTC forecast cycle during the 1999-2000 Florida cool season. 

Variable 

Config. 

RMS Error 

Bias 

Notable Errors 

Temp 

4-grid 

1 to 4 

-3 to l 

• 

Slightly larger errors occur in the 3-grid 
forecasts below 900 mb. 

(°C) 

3-grid 

1 to 4.5 

-4 to 1 

• 

All other errors are very similar. 





• 

A larger error occurs in the 4-grid forecasts 

Dew 

4-grid 

1 to 8 

-4 to 2 


below 900 mb, due to greater moist bias. 

Point 



• 

Both configurations tend to have a dry bias 

(°C) 

3-grid 

1 to 8 

-3 to 2 

• 

above 900 mb. 

Largest errors are in the 600-800 mb layer. 




-15 to 25 

• 

Both forecasts have the smallest errors at 

Wind 

4-grid 

10 to 55 


upper levels and largest errors at low levels. 

Direction 




• 

Largest RMS error differences are -10-1 5° 

(deg.) 

3 -grid 

10 to 55 

-15 to 20 


between 800-1000 mb. 

Wind 

4-grid 

2.5 to 6 

-2.5 to 2 

• 

Errors increase with height in both forecasts. 

Speed 
(m s') 



• 

The 3 -grid forecasts have a more negative 

3-grid 

2.5 to 5.5 

-3.5 to 1.5 


bias in the upper levels at the 1 1-h forecast. 
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1 200 UTC 3/4-grid Cycle Temperature (C) for 1 .8-m towers 
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Figure 4.8. A meteogram plot that displays a comparison between the 1200 UTC forecast cycle surface 
temperature errors (°C) from the 4- and 3-grid RAMS configurations during the 1999-2000 Florida cool season. 
Surface temperatures are verified at the 1.8-m level of the KSC/CCAFS wind tower network. Parameters 
plotted as a function of forecast hour are a) mean observed, mean 4-grid forecast, and mean 3 -grid forecast 
temperature, b) RMS error, c) bias, and d) error standard deviation (SD). The plotting convention is a solid line 
for the 4-grid forecasts, dot-dashed line for the 3-grid forecasts, and a dashed line for observed values. 
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Figure 4.10. A meteogram plot that displays a comparison between the 1200 UTC forecast cycle near- 
surface wind direction errors (degrees) from the 4-grid and 3-grid RAMS configurations during the 1999-2000 
Florida cool season. Wind direction is verified at the 16.5-m level of the KSC/CCAFS wind tower network. 
Parameters plotted as a function of forecast hour are a) RMS error and b) bias. The plotting convention is a 
solid line for the 4-grid errors and a dot-dashed line for the 3-grid errors. 


b. Upper-level Errors from 1200 UTC Cycle (XMR rawinsonde) 

As in Section 4. 1 .3.2, the XMR soundings taken at 23 15, and 1 1 1 5 UTC are used to verify temperature and 
dew point temperature forecasts for the 1 1-h, and 23-h forecasts, respectively. The 50-MHz DRWP is not used 
to verify winds because of the unreliable data during the 1999-2000 cool season. This brief discussion will only 
examine error statistics at XMR for the 1 1-h forecast valid at 2300 UTC. 

1. Temperature and dew point temperature 

Similar to the surface error statistics, only minor differences occur in the upper-level temperature and dew 
point temperature error statistics. Figure 4. 1 1 shows that the vertical profiles of the temperature RMS error and 
bias for the 4-grid and 3-grid forecasts generally exhibit small differences. Both configurations contain a low- 
level cold bias below 600 mb, reaching a maximum at about 975 mb (Fig. 4.1 lc). The greatest differences in 
RMS error occur above 800 mb where the 3 -grid errors are about 0.5°C smaller than the 4-grid errors (Fig. 
4.11b), resulting from differences in the random error component (SD in Fig. 4.1 Id). The temperature error 
profiles do not change substantially by 23 h in either model configuration (not shown). 

The profiles of dew point temperature errors indicate that the 4-grid RAMS forecasts tend to have a greater 
moist bias at the lowest levels below 950 mb whereas the 3-grid forecasts tend to have larger RMS errors 
between 600-700 mb (not shown). Both plots are quite noisy with widely changing errors with height. The 
largest errors are found between 600-800 mb in both forecasts and the patterns are generally quite similar (not 
shown). 

2. Winds 

Once again, only minor differences occur between the 4-grid and 3 -grid error statistics of upper-level winds. 
The wind direction RMS errors in both RAMS configurations decrease with height at the 1 1-h and 23-h forecast 
times (not shown). These errors are typically within 10° of each other at all levels. The wind speed errors have 
the greatest variation at upper levels where the errors are largest in both model configurations; however, these 
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1 200 UTC 3/4-grid Cycle 1 1 -h Wind Spd (m/s) at XMR 



Figure 4. 12. Vertical profiles of wind speed errors (m s 1 ) at XMR for the 11-h forecast displaying a 
comparison between the 4- and 3-grid configurations of RAMS from the 1200 UTC forecast cycle during the 
1999-2000 Florida cool season. Parameters plotted as a function of pressure are a) mean observed, mean 4-gnd 
forecast, and mean 3-grid forecast wind speed, b) RMS error, c) bias, and d) error standard deviation (SD). The 
plotting convention is a solid line for the 4-grid forecasts, dot-dashed line for the 3 -grid forecasts, and a dashed 
line for observed values. 
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4.4. 2.3 Discussion of 4-grid/3-grid Comparison 
a . Surface Errors from 1200 UTC Cycle (KSC/CCAFS wind towers) 

1. Temperature and dew point temperature 

Unlike the 1999-2000 cool-season results, the 4-grid/3-grid comparison during the 2000 warm season yields 
some substantial differences between the model configurations. The most significant difference between the 
operational 4-grid and the 3 -grid forecasts is the much larger surface temperature and dew point temperature 
errors of the 3-grid configuration caused in part by a more substantial cold, dry bias in the 3-grid forecasts. The 
differences between the two forecasts begin during the daylight hours and continue to increase during the 
remaining time out to 24 forecast hours. Figure 4.13 shows the error comparison between the 4-grid and 3-grid 
forecast temperatures at the KSC/CCAFS wind towers. After 3 h, the magnitude of the 3-grid RMS error, bias, 
and SD all exceed the 4-grid forecasts, typically by 1-2°C or more. Whereas the 4-grid errors decrease 
somewhat in magnitude during the overnight hours after 12 h, the 3-grid forecast errors do not taper as 
significantly overnight. In fact, after 20 h, the 3-grid RMS error increases further due to the increase in random 
errors (Figs. 4.13b and d). 

The differences in dew point temperature errors are even more substantial than the temperature errors. 
Figure 4.14 illustrates the dramatic growth in the 3-grid dew point temperature errors after 2 h. Between 2-5 h, 
the dew point RMS error jumps from 2-8°C in the 3-grid forecasts while staying at about 2°C in the 4-grid 
forecasts (Fig. 4. 14b). This 3-grid RMS error is composed of a dry bias of -5° to -6°C (peaking at 5 h and 24 h 
in Fig. 4.14c) and a random error peaking at 6-7.5°C at 5 h, and again after 20 h (Fig. 4.14d). Meanwhile, the 
4-grid dew point temperature errors are considerably smaller during all forecast hours, with a maximum RMS 
error slightly greater than 2°C, a bias near -1°C, and a SD at about 2°C (Figs. 4.14b-d). The cause of these 
much larger errors in the 3 -grid forecasts is not known. 

2. Winds 

In general, the surface wind errors are quite similar in the 4-grid and 3-grid RAMS forecasts during the 
2000 warm season. The RMS errors in wind direction are nearly identical during the first 4 h and afterwards, 
the 3-grid RMS errors are generally 5—10° larger (Fig. 4.15a). The wind direction biases are virtually the same 
until 13 h, after which they deviate by 10-20° (Fig. 4.15b); however, this deviation after 13 h is small compared 
to the magnitude of the RMS error during these hours. The only substantial difference in the wind speed errors 
is a slightly larger positive wind speed bias by 0.5— 1.0 m s 1 in the operational 4-grid forecasts during the 
daylight hours (not shown). This difference in wind speed bias is caused by a greater easterly bias in the 4-grid 
forecasts (negative u-wind bias, also not shown). 
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Figure 4.14. A meteogram plot that displays a comparison between the 1200 UTC forecast cycle surface 
dew point temperature errors (°C) from the 4- and 3-grid RAMS configurations during the 2000 Florida warm 
season. Surface dew point temperatures are verified at the 1.8-m level of the KSC/CCAFS wind tower network. 
Parameters plotted as a function of forecast hour are a) mean observed, mean 4-grid forecast, and mean 3 -grid 
forecast dew point temperature, b) RMS error, c) bias, and d) error standard deviation (SD). The plotting 
convention is a solid line for the 4-gnd forecasts, dot-dashed line for the 3-grid forecasts, and a dashed line for 
observed values. 
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The only substantial differences in errors occurs below 900 mb where a larger RMS error by about lms' is 
found in the operational 4-grid forecasts (Fig. 4. 1 7b). This difference in RMS error is almost entirely attributed 
to a larger positive wind speed bias in the 4-grid forecasts compared to the 3-grid configuration (Fig. 4. 1 7c). 

The upper-level wind direction errors are within 0—10° of each other at all times and levels for both the 50- 
MHz DRWP and the XMR rawinsonde (not shown). Also not shown are the individual wind component errors, 
which have only minor differences throughout the troposphere. 

4.4.3 Summary of 4-grid/3-grid Error Comparison 

The collective results from the 4-grid/3-grid comparison during the 1999-2000 cool, and 1999 and 2000 
warm seasons suggest the following about the RAMS model in east-central Florida: 

• Running a higher-resolution configuration of RAMS results in a significant improvement in the surface 
temperature and moisture error statistics during the warm season, but not during the cool season. 

• The error comparison does not yield a noticeable improvement in the surface and upper-level wind 
error statistics during both seasons. 

• During the warm season, the higher resolution configuration of RAMS tends to over predict wind 
speeds at the surface and lower levels of the atmosphere. 

Additional subjective evaluations in Section 5 will help to illustrate the total benefit of running the higher 
resolution, operational configuration of RAMS, particularly with respect to sea-breeze forecasts within the 
KSC/CCAFS wind-tower network. 
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1200 UTC 3/4-grid Cycle 10-h Wind Spd (m/s) at XMR 




Figure 4.17. Vertical profiles of wind speed errors (m s' 1 ) at XMR for the 10-h forecast displaying a 
comparison between the 4- and 3-grid configurations of RAMS from the 1200 UTC forecast cycle during the 
2000 Florida warm season. Parameters plotted as a function of pressure are a) mean observed, mean 4-grid 
forecast, and mean 3-grid forecast wind speed, b) RMS error, c) bias, and d) error standard deviation (SD). The 
plotting convention is a solid line for the 4-grid forecasts, dot-dashed line for the 3-grid forecasts, and a dashed 
line for observed values. 
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(Fig. 4.18b). At all other forecast hours, the RMS errors are nearly identical, generally 2-3°C. The larger 
RAMS RMS errors are attributed to the marked daytime cold bias (Fig. 4.18c) and a larger random error 
component as well (Fig. 4.18d). Unlike the meso-Eta evaluation results in Nutter and Manobianco (1999), the 
Eta model exhibits a warm bias of about 1— 2°C at most forecast hours, but the magnitude of the Eta warm bias is 
significantly less than the magnitude of the RAMS daytime cold bias. However, the Eta retains its warm bias at 
night while the RAMS bias diminishes to zero by 17 h (Fig. 4.18c). 

Both models generally have a slight moist bias in forecast dew point temperature as indicated by Figure 
4.19. During all 24 forecast hours, the RAMS and Eta models have nearly identical RMS errors and biases 
(Figs. 4. 19b-c). The most notable differences are found in the random error between 12-24 h. The RAMS SD 
is about 1-2°C larger than the Eta during these forecast hours (Fig. 4. 19d). Other than this minor difference, the 
errors are nearly the same during all forecast hours. 

h. Winds 

The major difference in the wind errors is due to a substantial positive bias in the Eta wind speed forecasts 
at 10 m. Since about November 1999, NCEP has experienced a post-processing problem that resulted in a 
significant positive wind speed bias in the Eta point forecasts, especially for stations at low elevations near sea 
level. This problem has resulted in wind speed biases on the order of 3-4 ms in the Eta point forecast at TTS 
for both the 1999-2000 cool and 2000 warm season. These wind speed biases are not representative of the 
actual Eta forecasts since this problem is purely a post-processing issue. Because of this problem in the Eta 
wind speed forecasts, the wind speed error plots will not be shown in this report. Please refer to 

httpw7www.emc.ncep.noaa.gov/mmb/research/nearstc/statsbYvblmod.html for additional information and bias 

plots for the Eta 48-h forecasts. 

The wind direction errors show that RAMS generally has slightly larger RMS errors by about 5-15°, mainly 
after 3 h (Fig. 4.20a). The most significant differences in wind direction RMS errors occur between 18-21 h 
(overnight hours). The biases indicate that the Eta model typically has a near zero or small positive bias 
whereas RAMS experiences a negative bias during the first 8 hours (Fig. 4.20b). After 8 h, the RAMS and Eta 
biases are very similar. The magnitudes of the bias in both models are small relative to the total error and thus, 
do not indicate any substantial systematic error in wind direction. 

The u-wind plots in Figure 4.2 1 show the tendency for RAMS to develop more of an easterly bias (negative 
u-wind bias) compared to the Eta model during the 1999-2000 cool season. Between 9—24 h, RAMS has about 
a -1 ms’ 1 u-wind bias whereas the Eta model has only a slight negative bias (Fig. 4.21c). The much larger RMS 
error and SD in the Eta model are indicative of the wind speed problem mentioned above (Figs. 4.21b and d). 
The v-wind biases are near zero for both the RAMS and Eta models (not shown). 
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Figure 4.19. A meteogram plot that displays a comparison between the 1200 UTC forecast cycle surface 
dew point temperature errors (°C) from the RAMS operational configuration and the Eta model during the 1999- 
2000 Florida cool season. Surface dew point temperatures are verified at TTS only. Parameters plotted as a 
function of forecast hour are a) mean observed, mean RAMS forecast, and mean Eta forecast dew point 
temperature, b) RMS error, c) bias, and d) error standard deviation (SD). The plotting convention is a solid line 
for the RAMS forecasts, dot-dashed line for the Eta model, and a dashed line for observed values. 
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Figure 4.21. A meteogram plot that displays a comparison between the 1200 UTC forecast cycle u-wind 
component errors (ms 1 ) from the RAMS operational configuration and the Eta model during the 1999-2000 
Florida cool season. The u-wind component is verified at TTS only. Parameters plotted as a function of 
forecast hour are a) mean observed, mean RAMS forecast, and mean Eta forecast u-wind, b) RMS error, c) bias, 
and d) error standard deviation (SD). The plotting convention is a solid line for the RAMS forecasts, dot- 
dashed line for the Eta model, and a dashed line for observed values. 
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The biases of the u- and v-wind components are shown in Figure 4.25 and indicate that the Eta model has a 
substantial southerly and easterly bias compared to RAMS. The magnitudes of the Eta point forecast biases are 
likely exaggerated due to the high wind speed bias resulting from the post-processing problem. Nonetheless, the 
Eta model exhibits an easterly bias between forecast hours 0 and 18 (Fig. 4.25a), and a southerly bias after the 6- 
h forecast (Fig. 4.25b). The RAMS u-wind plot shows a small bias of 0.5-1.0 m s' 1 between 5-14 h (Fig. 4.25a) 
and virtually no bias in the v-wind component (Fig. 4.25b). Thus, at the very least, the RAMS point forecasts at 
TTS result in an improvement in the v-wind component bias that occurs in the Eta model. 
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Figure 4.22. A meteogram plot that displays a comparison between the 1200 UTC forecast cycle surface 
temperature errors (°C) from the RAMS operational configuration and the Eta model during the 2000 Florida 
warm season. Surface temperatures are verified at TTS only. Parameters plotted as a function of forecast hour 
are a) mean observed, mean RAMS forecast, and mean Eta forecast temperature, b) RMS error, c) bias, and d) 
error standard deviation (SD). The plotting convention is a solid line for the RAMS forecasts, dot-dashed line 
for the Eta model, and a dashed line for observed values. 
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Figure 4.24. A meteogram plot that displays a comparison between the 1200 UTC forecast cycle surface 
wind direction errors (degrees) from the RAMS operational configuration and the Eta model during the 2000 
Florida warm season. Surface wind direction is verified at TTS only. Parameters plotted as a function of 
forecast hour are a) RMS error and b) bias. The plotting convention is a solid line for the RAMS errors and a 
dot-dashed line for the Eta model errors. 
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Figure 4.25. A meteogram plot that displays a comparison between the 1200 UTC forecast cycle u- and v- 
wind component biases (m s *) from the RAMS operational configuration and the Eta model during the 2000 
Florida warm season. The wind component biases are verified at TTS only. Parameters plotted as a function of 
forecast hour are a) u-wind bias, and b) v-wind bias. The plotting convention is a solid line for the RAMS 
forecasts and a dot-dashed line for the Eta model. 
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5. Subjective Evaluation Results 

This section presents results from an extensive number of subjective verification experiments for both the 
1999-2000 cool, and 1999 and 2000 warm seasons. The 1999-2000 cool-season subjective verification results 
include: 


• Frontal passages at seven east coast stations in Section 5.1.1. 

• Cool-season precipitation verification in Section 5.1.2. During the 1999-2000 cool season, very few 
significant precipitation events occurred and thus. Section 5.1.2 includes only a brief summary of the 
events that occurred along with two precipitation examples. 

• Low-level temperature inversions at the XMR rawinsonde in Section 5.1.3. 

The warm season results also include three components: 

• Sea-breeze verification at the KSC/CCAFS wind towers and at TTS for 1999 and 2000 forecasts 
presented in Section 5.2.1. 

• Precipitation validation on RAMS grid 4 for the 2000 months shown in Section 5.2.2. 

• Thunderstorm initiation verification for the 2000 months given in Section 5.2.3. 

Even though the ERDAS RAMS interim report contained the 1999 warm season sea-breeze verification 
results, the AMU evaluated these sea-breeze forecasts again for two reasons. First, the AMU modified the 
evaluation technique to include all available archived forecast data for both the 1999 and 2000 warm seasons 
compared to data only during AMU working days, as was done in the interim report on the 1999 data. Second, 
by performing the evaluation for all available archived forecasts, the AMU could build a database of sea-breeze 
verification statistics for two Florida warm seasons, thereby increasing the robustness of the results. It is 
important to note that the 1999 database contains forecasts during the months of May-August whereas the 2000 
warm season database contains forecasts for May-September. 

The precipitation and thunderstorm initiation verifications are performed on the 2000 warm season forecasts 
only, as tasked in the ERDAS RAMS extension evaluation. The precipitation verification closely follows the 
technique used in the ERDAS RAMS interim report, but with a few modifications (refer to Section 2.2.2. 2 and 
Case 2000). As with the sea-breeze evaluation, the precipitation verification was modified from the technique 
used for the 1999 warm season to include all available archived forecasts during the 2000 warm season in order 
to maximize the verification data base. However, the appropriate WSR-74C reflectivity fields for the 
precipitation verification were not archived until late May 2000. Thus, the precipitation verification is 
performed only for the months of June-September 2000. The thunderstorm initiation verification was 
conducted for all 2000 warm season months (May-September). 

5.1 1999-2000 Cool Season 

The results from the 1999-2000 Florida cool-season subjective verification are presented in this section. 
All three components of the cool-season subjective verification were conducted from November 1999 to March 
2000. Refer to Section 2.2.1 for the methodology used in each experiment. 

5.1.1 Verification of Fronts 

During the five cool-season months, the AMU documented all occurrences of any type of observed frontal 
discontinuities (wind shifts, temperature, or dew point temperature gradients). Graphical traces (meteograms) of 
observed and forecast hourly temperature, dew point temperature, wind direction, wind speed, and pressure 
observations were examined at seven selected surface stations in the Florida peninsula [Jacksonville (JAX), 
Daytona Beach (DAB), the Shuttle Landing Facility (TTS), Melbourne (MLB), Vero Beach (VRB), West Palm 
Beach (PBI), and Miami (MIA)]. Figure 5.1 shows a sample meteogram from 22 December 1999 illustrating 
the evolution of forecast and observed variables during the passage of a weak front at TTS. Note the smooth 
transition in the forecast wind shift compared to the observed wind shift in Figure 5. lb. From 1400-1500 UTC, 
the observed wind direction changes sharply from southwesterly (~240°) to northwesterly (> 300°). Meanwhile, 
the RAMS forecast has a very gradual shift from south-southwesterly winds at 1400 UTC to northwesterly by 
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Figure 5.2 shows an example of a RAMS forecast at JAX during the passage of a strong cold front. In this 
instance, RAMS did a particularly poor job in forecasting the intensity of this cold front. The observed cold 
front was accompanied by a notable increase in wind speed from 3 to 7 m s 1 (Fig. 5.2a), a sharp wind shift from 
southerly to northwest (Fig. 5.2b), a 3-h temperature drop of 1 1°C (from 23 to 12°C in Fig. 5.2c), and a more 
gradual decrease in dew point temperature (Fig. 5. 2d). In this case the frontal intensity increased dramatically 
due to evaporational cooling caused by post-frontal precipitation that fell over southern Georgia and northern 
Florida (not shown). 
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Figure 5.2. A plot of surface variables at JAX used for verifying RAMS forecast frontal 
passages/discontinuities during the 1999—2000 Florida cool season. This example shows the RAMS forecast 
during the passage of a strong cold front on 30-31 January 2000. The variables plotted are a) wind speed, b) 
wind direction, c) temperature, and d) dew point temperature. 
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Table 5.2 summarizes the seven most substantial precipitation events during the 1999-2000 cool season. 
Based on these events, RAMS performed with variable accuracy when forecasting cool-season precipitation 
events across the Florida peninsula. RAMS generally did not accurately predict the timing and location of pre- 
frontal rain bands, but could handle the rain bands associated with the actual cold front better. With some 
events such as 17 December 1999 and 31 January 2000, RAMS did not predict any substantial precipitation 
despite the occurrence of fairly widespread observed rainfall. In the 2 November 1999 cold front and associated 
squall line, the timing of the frontal passage was substantially in error and thus, the rainfall was not predicted 
well at all. In the 12 March and 27 March 2000 frontal events, RAMS did not predict the significant convection 
that occurred over southern Florida, but accurately predicted the lighter precipitation over central Florida. 

Figure 5.3 shows an example of how RAMS missed the pre-frontal rain band during the 24 January 2000 
strong frontal passage over the Florida peninsula. This figure also illustrates the accurate prediction of the 
frontal band; however, most of the precipitation fell with the pre-frontal squall line, so the forecast rainfall rates 
are much too high associated with the frontal passage. At 0900 UTC, a substantial amount of convection is 
prevalent across the Florida peninsula, especially concentrated over the southern and eastern portions (Fig. 
5.3a). RAMS does not indicate forecast precipitation in any portion of the grid-2 domain at this time. The 
squall line has cleared much of the peninsula by 1200 UTC when a second band of convection becomes 
prevalent across the northwestern portions of grid 2 (Fig. 5.3b). Between 1500-1800 UTC, the frontal band 
quickly moves into central Florida and weakens noticeably by 1800 UTC (Figs. 5.3c-d). At these times, RAMS 
predicts a fairly intense rain band associated with the second observed cloud band with rainfall rates exceeding 
40 mm h' 1 in some instances. However, only light precipitation occurred with the passage of the frontal band 
since most of the heaviest rainfall occurred with the pre-frontal rain band. This case shows how RAMS had 
difficulty in predicting pre-frontal squall lines during the 1999-2000 cool-season precipitation events. 
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Figure 5.3. The RAMS grid-2 forecast versus observed precipitation features during the 0000 UTC 24 
January 2000 forecast cycle. Enhanced GOES-8 infrared (IR) imagery is plotted along with contoured observed 
hourly precipitation across eastern and southern Florida (blue lines, units of mm h ') and forecast precipitation 
rate (dashed green lines, units of mm h '). Valid times on 24 January are: a) 0900 UTC (9-h forecast), b) 1200 
UTC (12-h forecast), c) 1500 UTC (15-h forecast), and d) 1800 UTC (18-h forecast). The dark red colors 
represent the coldest cloud-top temperatures and deepest convection. 


Figure 5.4 depicts a 4-h evolution of clouds and scattered rain showers in advance of a cold front during the 
afternoon hours of 11 March 2000. In this case, the 1200 UTC cycle of RAMS performed quite well in 
generating scattered shower activity during the afternoon hours across the eastern portions of the Florida 
peninsula. The timing of the forecast precipitation is accurate to the nearest hour and the location is slightly 
north and west of the observed precipitation. At 2000 UTC, scattered convection develops across southern and 
eastern Florida close to the Atlantic coast whereas RAMS develops showers just to the west of the observed rain 
areas (Fig. 5.4a). The rainfall continues to develop over the next 2-3 hours as the entire disorganized band 
pushes offshore to the north and east (Figs. 5.4b-d). Meanwhile, RAMS develops additional rainfall across 
central and portions of southern Florida in Figures 5.4b-d and these showers move northeast and consolidate 
over east-central Florida, to the north and west of the observed convection. The results from this case illustrate 
that RAMS can offer some utility in predicting scattered rainfall activity in advance of a cold front. 
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When RAMS successfully forecasts a low-level inversion, the model typically underestimates the intensity 
of the inversion by 2.5°C (bias of -2.5°C in Table 5.4). The RMS errors in forecast inversion depth and height 
of the inversion base are 202 m and 516 m, respectively (2nd and 3rd rows of Table 5.4). The model has a 
slight tendency to spread the inversion through a deeper layer than observed (59 m bias). Not directly indicated 
in these tables is the difficulty that RAMS demonstrated in its ability to consistently predict surface-based 
inversions. Many surface-based inversions were either not forecast by RAMS, or the predicted inversion 
occurred above the surface in the lowest 1 km. 


Table 5.3. A contingency table of the number of the combined 0000 
and 1200 UTC RAMS forecast versus observed occurrences of low-level 
temperature inversions at the CCAFS rawinsonde during the period of 
November 1999 to March 2000. The categorical scores derived from this 
table are shown below. 

0000/1200 UTC Cycles 

Observed Inversion 

No Observed Inversion 

Forecast Inversion 

86 

3 

No Forecast Inversion 

103 

15 

Probability of Detection: 
False Alarm Rate: 

0.46 Bias: 0.47 

0.03 CSI: 0.45 


Table 5.4. A summary of the root mean square (RMS) error and bias 
statistics of the RAMS forecast temperature inversion intensity (°C), depth of 
the inversion (m), and height of the inversion base (m), using the CCAFS 
rawinsonde from November 1999 to March 2000. 

Parameter 

RMS Error 

Bias 

Intensity (°C) 

4.1 

-2.5 

Depth (m) 

202 

59 

Height (m) 

516 

22 


5.2 1999 and 2000 Warm Seasons 

The AMU conducted an extensive sea-breeze verification for the 1999 and 2000 Florida warm seasons to 
determine the potential utility of using RAMS for surface wind forecast guidance during the warm season. In 
addition, the AMU benchmarked the skill of RAMS sea-breeze predictions to the NCEP Eta model point 
forecasts at TTS and compared the skill of the operational configuration to the coarser 3-grid configuration of 
RAMS. Furthermore, the AMU conducted a precipitation and thunderstorm initiation verification during the 
2000 warm season months. This section presents the results from the sea-breeze, precipitation, and the 
thunderstorm initiation verifications during the 1999 and 2000 warm seasons. 
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Figure 5.5. A plot of observed KSC/CCAFS 16.5-m wind flow on 18 August 2000 illustrating the 
interaction between local river and ocean breezes. The valid times for each panel are a) 1500 UTC, b) 1600 
UTC, c) 1 700 UTC, and d) 1 800 UTC. 


In Figures 5.7 and 5.8, graphical traces (meteograms) of wind direction and wind speed are used to verify 
subjectively the sea-breeze passage at each of the 12 KSC/CCAFS wind towers given in Figure 2.2b. In both 
figures, the 12 meteograms representing each of the 12 wind towers are arranged to follow the north-south, and 
east-west spatial layout of the wind towers in east-central Florida. For example, the meteograms of the 4 
westernmost wind towers of Figure 2.2b (819, 1012, 1007, and 1000) are arranged on the far left (west) portion 
of the figure. In addition, their north-south orientation is preserved and plotted top-bottom in Figures 5.7 and 
5.8. This plotting convention facilitated the subjective sea-breeze verification since these meteograms were 
examined for every day of 9 months worth of data. To determine an observed or forecast sea-breeze passage at 
an individual station, the wind direction is first examined to see if a shift from offshore to onshore occurs. Wind 
directions between the dashed lines on each meteogram of Figure 5.7 represent offshore winds whereas wind 
directions outside the dashed lines are onshore. Under light or easterly flow, the wind speed plots are also 
examined to identify the accompanying wind speed increase with the sea-breeze passage. 
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With the exception of the wind speeds being too strong, this case illustrates how remarkably well RAMS 
can predict the interaction between river and sea breezes during a light and variable wind regime. The 
development of the circulations and the timing of these wind features are excellent at each hour between 1500 
and 1800 UTC. Since the river and sea breezes often serve as the focal points for warm-season convective 
initiation, it is critical that a model such as RAMS can predict these local interactions at sufficiently high 
resolutions on a regular basis. The remaining portion of this section presents the collective statistics and skill 
scores from the 1999 and 2000 warm-season sea-breeze evaluations. 
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Figure 5.7. A meteogram plot of RAMS forecast versus observed wind direction (deg) for the 0000 UTC 
forecast cycle of 18 August 2000. Observed points are given by an asterisk whereas a triangle represents 
forecast values. The KSC/CCAFS towers used for verification (Fig. 2.2, and labeled above each graph) are 
arranged according to the spatial location in east-central Florida. The left/center/right-hand columns are the 
mainland/Merritt Island/coastal towers, and each are arranged in a north-south orientation from the top-bottom. 
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and 1200 UTC forecasts are statistically significant. Refer to Appendix D for more details regarding the 
statistical significance tests. 

In the instances when a correct yes forecast of a sea breeze occurred, the timing errors were determined at 
each of the wind towers during the 9-month evaluation period. Table 5.7 summarizes the timing error statistics 
for all the correct yes forecasts of a sea breeze for both the 0000 and 1200 UTC cycles. In general, the RMS 
error ranges from 1.5-2. 1 h for each category of wind towers. The errors are smallest at the coastal towers and 
largest at the mainland towers, but the variation is less than 0.5 h, which is smaller than the data sampling rate of 
once per hour. In all instances the bias is -0.2 or -0.3 h, which is negligible compared to the sampling rate. 


Table 5.5. Contingency tables of the occurrence of the operational RAMS forecast 
versus observed sea breeze, verified at each of the 12 selected KSC/CCAFS towers of 
Figure 2.2b during the 1999 and 2000 Florida warm seasons. 

0000 UTC Forecast Cycle 

Observed Sea Breeze 

No Observed Sea Breeze 

Forecast Sea Breeze 

1381 

261 

No Forecast Sea Breeze 

228 

599 

1200 UTC Forecast Cycle 

Observed Sea Breeze 

No Observed Sea Breeze 

Forecast Sea Breeze 

1575 

293 

No Forecast Sea Breeze 

34 

567 


Table 5.6. Categorical and skill scores of RAMS forecast versus observed sea 
breeze during the 1999 and 2000 Florida warm seasons, associated with the contingencies 
in Table 5.5. 

Parameter 

0000 UTC Forecast Cycle 

1200 UTC Forecast Cycle 

Probability of Detection 

0.86 

0.98 

False Alarm Rate 

0.16 

0.16 

Bias 

1.02 

1.16 

Critical Success Index 

0.74 

0.83 

Heidke Skill Score 

0.56 

0.69 


Table 5.7. A summary of timing error statistics for the May-August 1999 and 
May-September 2000 evaluation periods are given for the subjective sea breeze 
verification performed for the 12 KSC/CCAFS tower locations of Figure 2.2b. The RMS 
error and bias are shown in units of hours for the 0000 UTC and 1200 UTC forecast runs. 

Location 

Statistic 

0000 UTC Cycle 

1200 UTC Cycle 


RMS Error 

1.8 

1.5 

Coastal Towers 


-0.3 


Bias 

-0.3 


RMS Error 

1.9 

1.7 

Merritt Island Towers 


-0.3 

-0.2 

Bias 


RMS Error 

2.1 

1.9 

Mainland T owers 


-0.3 

-0.2 


Bias 
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Finally, Tables 5.12 and 5. 13 compare the skill between the 4-grid and 3-grid RAMS forecasts only at the 
mainland wind towers west of the Indian river. This group of wind towers exhibited the greatest discrepancy in 
skill between the 4-grid and 3-grid RAMS forecasts during the 2000 warm season. Table 5.12 shows that there 
were 70 more missed ECSB forecasts in the 3-grid compared to the 4-grid RAMS. As a result, the POD is 14% 
higher in the 4-grid forecasts, resulting in a 9% higher CSI and 1 1% higher HSS (Table 5.13). Contrary to the 
comparison of wind objective error statistics, these results suggest that the 4-grid configuration better resolves 
the interactions between the river and sea breezes, providing a dramatic improvement in the ECSB forecasts. 
The 5-km horizontal resolution of RAMS grid 3 is simply not sufficient to resolve river breeze circulations 
adequately since theoretically, it cannot resolve features whose wavelengths are less than 20 km (4 times the 
horizontal grid spacing). Meanwhile, the 1.25-km grid can resolve features with wavelengths as small as 5 km, 
which is comparable to or smaller than the scale of river-breeze circulations [CHECK FOR REFERENCE], 

These results also show that objective error statistics alone are not sufficient to evaluate the potential utility 
that a high-resolution model can provide to forecasters. The warm-season objective comparison between the 4- 
grid and 3-grid configurations of RAMS showed little difference in the errors during the 2000 warm season 
(refer to Section 4.4.2). However, the phenomenological verification presented in this section clearly shows that 
the higher resolution RAMS configuration has greater skill in predicting the ECSB over the course of a 5-month 
warm season. The accurate prediction of phenomenological features are quite important in determining the 
added value of a modeling system for everyday forecasting at the 45 WS and dispersion modeling for the 45 
SW/SE. 


Table 5.10. Contingency tables of the occurrence of the 1200 UTC 4-grid and 3 -grid 

configurations of RAMS versus observed sea breeze, verified at each of the 1 2 selected 
KSC/CCAFS towers of Figure 2.2b during the 2000 Florida warm season. 

1200 UTC RAMS 4-grid 

Observed Sea Breeze 

No Observed Sea Breeze 

Forecast Sea Breeze 

767 

135 

No Forecast Sea Breeze 

15 

286 

1200 UTC RAMS 3-grid 

Observed Sea Breeze 

No Observed Sea Breeze 

Forecast Sea Breeze 

719 

126 

No Forecast Sea Breeze 

63 

295 


Table 5.1 1. Categorical and skill scores of the 1200 UTC RAMS 4-grid and 3-grid 
configurations versus observed sea breeze during the 2000 Florida warm season, 
associated with the contingencies of Table 5.10. 

Parameter 

1200 UTC RAMS 4-grid 

1200 UTC RAMS 3-grid 

Probability of Detection 

0.98 

0.92 

False Alarm Rate 

0.15 

0.15 

Bias 

1.15 

1.08 

Critical Success Index 

0.84 

0.79 

Heidke Skill Score 

0.71 

0.64 
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Even though RAMS dramatically outperformed the Eta model during the 0000 UTC forecast cycle, the 
same cannot be said for the 1200 UTC cycle. Tables 5.16 and 5.17 indicate that RAMS still correctly predicts 
the occurrence of the ECSB at a higher percentage than the Eta model; however, RAMS also has a significantly 
higher FAR than the Eta model. As a result, the CSI and HSS of Table 5.17 are only marginally better in 
RAMS. In fact, neither the CSI nor the HSS differences are statistically significant above the 86% confidence 
interval (see Appendix D). Meanwhile, the higher FAR in RAMS is statistically significant compared to the Eta 
model, suggesting that RAMS has a tendency to overpredict the occurrence of the ECSB in the 1200 UTC cycle 
compared to the Eta model. Overall though, RAMS clearly demonstrates that it has the ability the better detect 
the occurrence of the ECSB at TTS during the Florida warm season, especially during the 0000 UTC forecast 
cycle. These results indicate that, despite the comparable or slightly better objective error statistics in the Eta 
model, the phenomenological verification of the ECSB improves over the Eta model when running the RAMS 
model with fine horizontal grid spacing such as in the current configuration. 


Table 5.14. Contingency tables of the occurrence of the 0000 UTC operational 
RAMS and Eta forecast versus observed sea breeze, verified at TTS for the 1999 and 2000 
warm season months. 


0000 UTC RAMS 


Forecast Sea Breeze 
No Forecast Sea Breeze 


0000 UTC Eta 


Forecast Sea Breeze 
No Forecast Sea Breeze 


Observed Sea Breeze 


No Observed Sea Breeze 



Observed Sea Breeze 


No Observed Sea Breeze 



Table 5.15. Categorical and skill scores of the 0000 UTC RAMS and Eta forecast 

versus observed sea breeze during the 1999 and 2000 Florida warm seasons, associated 

with the contingencies of Table 5.14. 


Parameter 

0000 UTC RAMS 

0000 UTC Eta 

Probability of Detection 

0.83 

0.53 

False Alarm Rate 

0.15 

0.13 

Bias 

0.98 

0.61 

Critical Success Index 

0.72 

0.49 

Heidke Skill Score 

0.61 

0.38 


Table 5.16. Contingency tables of the occurrence of the 1200 UTC operational 
RAMS and Eta forecast versus observed sea breeze, verified at TTS for the 2000 warm 
season months only. 


1200 UTC RAMS 


Forecast Sea Breeze 
No Forecast Sea Breeze 


1200 UTC Eta 


Forecast Sea Breeze 
No Forecast Sea Breeze 


Observed Sea Breeze 


No Observed Sea Breeze 


Observed Sea Breeze 


No Observed Sea Breeze 
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Figure 5.9. Observed WSR-74C hourly reflectivity from a 2000-ft Constant Altitude Plan Position 
Indicator on 25 July 2000. Valid times are: a) 1800 UTC, b) 1900 UTC, c) 2000 UTC, and d) 2100 UTC. 
Orange and red colors represent reflectivity greater than 32 dBZ used to classify 5 mm h 1 rainfall rate intensities 
for verifying RAMS forecasts. 
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To obtain the seasonal precipitation verification statistics, the verification methodology described above for 
the specific case was repeated for both the 0000 and 1200 UTC forecast cycles during all days in the 
June-September 2000 time frame. The precipitation verification statistics are summarized in three ways: 

• Occurrence of precipitation as a function of zone at any time within the 9-h window of 1500—2300 
UTC. This test determines how well RAMS predicted precipitation occurrence within each zone 
anytime between the hours of 1500-2300 UTC. 

• Timing and location as a function of 1-h, 2-h, and 3-h verification time windows (bins). The 1-h bins 
are the most stringent verification, since the observed and forecast instantaneous rainfall must occur in 
the same zone at the same hour for a hit. The 2-h verification bins are valid for 1600—1700, 
1800-1900, 2000-2100, and 2200-2300 UTC. In these 2-h bins, a hit occurs when RAMS predicts 
rainfall in either one of the two hours and rainfall is observed at either hour. Finally, the 3-h windows 
are valid from 1500-1700, 1800-2000, and 2100-2300 UTC, and a hit occurs when forecast and 
observed rain occurs in one or more of the 3-h times. 

• Timing and location as a function of hour within the verification window. In this validation, the 
performance of RAMS precipitation forecasts is verified at each hour of the 1500-2300 UTC 
verification window. For the 2- and 3-h bins, the precipitation statistics are presented for the first hour 
of the valid verification time frame. 

For each scenario described above, contingency tables were developed and corresponding categorical and skill 
scores were calculated. 

5.2.2 . 1 Occurrence of Daily Precipitation in each Grid-4 Zone 

Figure 5.1 1 shows the categorical and skill scores for daily 0000 and 1200 UTC RAMS precipitation-rate 
predictions in each zone of grid 4 between 1500-2300 UTC. Based on these charts, RAMS has the highest skill 
in predicting daily precipitation over the inland zones 1-3 compared to the coastal zones 4—6. The POD is close 
to 0.8 on each of the inland zones whereas the POD ranges from about 0.45 to 0.70 over the coastal zones (Fig. 
5.11a). RAMS generally has a lower FAR over the inland zones compared to the coastal regions (Fig. 5.1 lb). 
As a result, the CSI (analogous to precipitation threat score) in Figure 5.1 lc is highest at nearly 0.70 over zones 
1-3, and lowest over the coastal zones 4-6, ranging from 0.35-0.55. The bias plots in Figure 5.1 Id indicate 
that RAMS has a tendency to underpredict the daily occurrence of precipitation, especially in zones 1, 4, and 6 
of the 1200 UTC forecast cycle, and in all zones of the 0000 UTC cycle. 

Not only does RAMS predict daily precipitation with higher skill over inland regions, it also exhibits 
greater skill in the 1200 UTC versus 0000 UTC forecast cycle. In every zone of grid 4, the 1200 UTC cycle has 
a higher POD, CSI, and bias (Fig. 5.11). The FAR is not dramatically different in any grid-4 zones. It is also 
interesting to note that zone 6 experiences the poorest forecast skill in both the 0000 and 1200 UTC cycles. 
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Figure 5.12. Categorical scores for the RAMS 1200 UTC precipitation forecasts plotted as a function of 
the 6 zones on grid 4 and time verification window. The scores shown in this figure include a) the Probability of 
Detection (POD), b) the False Alarm Rate (FAR), c) the Critical Success Index (CSI), and d) the Bias. The 
POD, FAR, CSI, and bias are each shown for 1-h, 2-h, and 3-h time verification windows. The lightest shade is 
the 1-h time window whereas the darkest shade is the 3-h time verification window. The 1-h window represents 
each instantaneous hourly time from 1500-2300 UTC. The 2-h windows are the instantaneous hours valid at 
1600-1700, 1800-1900, 2000-2100, and 2200-2300 UTC. The 3-h windows are the instantaneous hours valid 
at 1500-1700, 1800-2000, and 2100-2300 UTC. 

5. 2.2.3 Timing and Location of Precipitation at each Hour 

The third and final examination of the precipitation verification statistics is the timing and location 
validation for all six zones as a function of hour of the day, shown in Figure 5.13. For the 1-h verification bins 
in Figure 5.13a, 1700 UTC is the only hour when the POD is greater than the FAR, representing the peak of 
hourly CSI values. The lowest amount of skill occurs at 1500 UTC and especially after 2000 UTC, when the 
CSI drops well under 0.20 between 2100-2300 UTC. This reduction in skill is especially prevalent in the 2-h 
and 3-h verification bins in Figures 5.13b-c. The POD peaks at 1800 UTC in the 2-h bins and decreases sharply 
over the next 4 hours in conjunction with a sharp increase in the FAR during the same times (Fig. 5.1 3b). In the 
3-h verification bins, the POD is nearly constant for the 3-h periods that begin at 1500 and 1 800 UTC, and then 
sharply decreases in the final 3-h time frame (Fig. 5.13c). 

The poor skill during the late afternoon and evening hours are likely caused by the model s inability to 
accurately predict the evolution of outflow boundaries and their interactions. In fact, one erroneously predicted 
storm can result in many subsequent erroneous storms that develop in the model along the incorrect placement 
of outflow boundaries. As a result, the model’s skill decreases dramatically by the late afternoon hours as 
suggested by Figure 5.13. These model results would most likely improve dramatically with a more 
sophisticated data assimilation scheme that ingests continuous observational data such as WSR-88D and GOES- 
8 brightness temperature data. Furthermore, a data assimilation scheme where analyses and model forecasts are 
cycled much more frequently than the current scheme (every 12 hours) should noticeably improve the short- 
range precipitation forecasts. 
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Verification of Daily Thunderstorm Initiation 


The last portion of the subjective verification that the AMU conducted for the 2000 warm season is the 
daily thunderstorm initiation verification. Following the observed and model thunderstorm definitions outlined 
in Section 2. 2. 2. 3, the AMU developed a seasonal spreadsheet for May-September to tally the thunderstorm 
initiation results for both the 0000 and 1200 UTC forecast cycles. 


14 May 2000, 1700 UTC 


14 May 2000, 5-h forecast at 1700 UTC 



W/// 
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14 May 2000, 1800 UTC 


14 May 2000, 6-h forecast at 1800 UTC 


Figure 5.14. Hourly GOES-8 visible satellite imagery and surface winds overlaid with CGLSS cloud-to- 
ground lightning strikes (denoted by a colored ‘X’ in a, b, and c) compared to RAMS forecast surface winds, 7- 
km vertical velocity (m s' 1 , shaded according to the scale provided), and surface instantaneous precipitation rate 
(mm h' 1 ) (d, e, and 0 on 14 May 2000 over the area of RAMS grid 4. Valid times for the observed plots are a) 
1700 UTC, b) 1800 UTC, and c) 1900 UTC and the corresponding RAMS forecast plots at d) 1700 UTC, e) 
1800 UTC, and f) 1900 UTC. 
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5. 2. 3. 1 Occurrence of Thunderstorm Days Anywhere on Grid 4 

According to Tables 5.18 and 5.19, the 1200 UTC RAMS forecast cycle predicted the occurrence of a 
thunderstorm day on grid 4 much better than the 0000 UTC cycle. In the 1200 UTC cycle, the number of 
correctly predicted thunderstorm days is higher than the 0000 UTC cycle in combination with fewer missed 
forecasts (Table 5.18). As a result, the 1200 UTC cycle has a higher POD than the 0000 UTC cycle (Table 
5.19); however, the 1200 UTC cycle does have a slightly greater tendency towards false alarms and over- 
predicting thunderstorm days as indicated by the slightly higher FAR and a bias greater than 1.0 (Table 5.19). 
Nonetheless, the overall skill in Table 5.19 indicates that the 1200 UTC cycle realizes a 21% higher HSS than 
the 0000 UTC cycle. The statistical significances of these differences were not determined. 


Table 5.18. Contingency tables of the occurrence of RAMS predicted versus 
observed thunderstorms anywhere on grid 4, verified each day between 1500—2300 UTC 
during the 2000 Florida warm season. 

0000 UTC Forecast Cycle 

Observed T-storms 

No Observed T-storms 

Forecast T-storms 

36 

11 

No Forecast T-storms 

35 

45 

1200 UTC Forecast Cycle 

Observed T-storms 

No Observed T-storms 

Forecast T-storms 

62 

21 

No Forecast T-storms 

9 

35 


Table 5.19. Categorical and skill scores of RAMS forecast versus observed 
thunderstorms anywhere on grid 4, associated with the contingencies of Table 5.22. 

Parameter 

0000 UTC Forecast Cycle 

1200 UTC Forecast Cycle 

Probability of Detection 

0.51 

0.87 

False Alarm Rate 

0.23 

0.25 

Bias 

0.66 

1.17 

Critical Success Index 

0.44 

0.67 

Heidke Skill Score 

0.30 

0.51 


5.2.3. 2 Timing and Spatial Accuracy of Forecast Thunderstorm Initiation 

In general, both forecast cycles are comparable in terms of the spatial accuracy, whereas the 1200 UTC 
cycle exhibits slightly more favorable results in the timing of thunderstorm initiation. The timing RMS errors of 
thunderstorm initiation anywhere on RAMS grid 4 were generally between 2-3 h for both forecast cycles 
whereas the bias was about 1 h in the 0000 UTC cycle and 0 h in the 1200 UTC cycle (not shown). The timing 
error statistics for thunderstorm initiation in each individual grid-4 zone did not exhibit any trends or organized 
patterns that favored specific zones. 

Table 5.20 summarizes the spatial and timing results of the RAMS forecast thunderstorm initiation for the 
0000 and 1200 UTC cycles. Spatially, both forecast cycles correctly predicted thunderstorm initiation in one or 
more zones about half the time (58% in 0000 UTC cycle and 46% in 1200 UTC cycle, Table 5.20). The slightly 
poorer performance of the 1200 UTC cycle could be attributed to the larger sample size of correctly-forecast 
thunderstorm days. In the timing accuracy, only 8% (19%) of the correctly predicted thunderstorm days 
experienced an exact initiation time to the nearest hour in the 0000 UTC (1200 UTC) cycle. Meanwhile, RAMS 
correctly predicted the hourly thunderstorm initiation time to within 3 hours of the observed time about 75% of 
all days for both forecast cycles (slightly higher in the 1200 UTC forecasts). Note that these timing accuracies 
in RAMS do not reflect off-hour predictions because forecast output was available only at the top of each hour. 
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Figure 5.15. The Probability of Detection (POD) and False Alarm Rate (FAR) for the RAMS 0000 and 
1200 UTC forecasts of the first daily thunderstorm occurrence in each zone of grid 4 during the hours of 
1500-2300 UTC. The 0000 UTC POD and FAR are shown in a) and b) respectively, and the 1200 UTC POD 
and FAR are shown in c) and d) respectively. The scores were determined by verifying hourly RAMS 
thunderstorm occurrences to the nearest 1 h, 2 h, 3 h, and for the entire daily verification period according to the 
scale provided. 
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1 200 UTC Exp/4-oper Cycle Temperature (C) for 1 .8-m tower 




Forecast Hour 


Figure 6.1. A meteogram plot of temperature errors (°C) for the experimental 1 200 UTC RAMS forecasts 
using Eta 0-h forecasts as a background field for the initial conditions, and 0— 24-h Eta forecasts as boundary 
conditions. The temperatures are verified at the 1.8-m level of the KSC/CCAFS wind-tower network. 
Parameters plotted as a function of forecast hour are a) mean observed, mean operational RAMS, and mean 
experimental RAMS forecast temperatures, b) RMS error, c) bias, and d) error standard deviation (SD). The 
labeling convention is a dashed line for observed values, a solid line for the operational RAMS, and a dot- 
dashed line for the experimental RAMS configuration. 


95 






7 . 


Recommendations for Improvements 


In this section, the AMU offers some recommendations for modifying and improving the existing ERDAS 
RAMS forecast and display system. In particular, the AMU highly recommends an improved graphics software 
package for routine analysis and display of RAMS forecasts. Based on the combination of the objective and 
subjective evaluations, RAMS clearly offers sufficient added value to short-range forecasts in support of the US 
Space Program. 

7.1 Improve Current Graphics Capabilities 

7.1.1 Replace Current RAMS Display 

According to the AMU’s assessment, the current graphics software within the MARSS system in the Range 
Weather Operations (RWO) is not a sufficient interface for displaying and analyzing RAMS forecasts in real- 
time. There are a number of deficiencies and problems with the display software that make it difficult for any 
user to efficiently access and analyze forecast data. Some of these deficiencies include: 

• Frequent and unpredictable hang-ups and crashes when displaying various combinations of forecast 
variables. 

• Occasional very slow updates when selecting new parameters or grids. 

• Inadequate control and insufficient contouring and labeling of scalar fields. 

• Very difficult to exit the RAMS interface cleanly without having to kill the existing session. 

Other problems may exist with the interface, but these difficulties alone make it quite difficult to use in an 
operational setting when fast and reliable graphical products are critical. 

One alternative to the current RAMS display software that the AMU recommends is the Visualization in 5 
dimensions (Vis5D) software. The five dimensions of Vis5D consist of the three spatial dimensions, a 
dimension in time for animation, and a dimension for displaying multiple variables. Several advantages to using 
Vis5D as a display software are listed below: 

• Vis5D is a visually appealing and easy-to-use display software that gives the user extensive control. 

• Vis5D can display horizontal or vertical slices, “isosurfaces” of a constant value in three-dimensions, 
overlay multiple variables, and can animate all combinations of displays in three-dimensional space. 

• A routine already exists within the ERDAS RAMS package to convert the RAMS forecast data into 
Vis5D format. 

• The software is freely available from the University of Wisconsin and can be easily installed onto the 
existing workstations in the RWO. 

• Vis5D has a shallow learning curve and is easy to use the very first time, so required training would be 
minimal. 

Another alternative graphical package is the General Meteorological Package (GEMPAK) software. 
Available from the Unidata division of the University Corporation for Atmospheric Research, GEMPAK has an 
extensive amount of capabilities for displaying and computing derived meteorological quantities from gridded 
forecast fields. As a diagnostic tool, GEMPAK may be more powerful than Vis5D; however, the learning curve 
for GEMPAK is steeper than Vis5D. Also, in a time-restricted environment, Vis5D would likely provide a 
snapshot of the forecast data in a shorter time period. 

7.1.2 Utilize Real-time Verification Graphical Tool 

In addition to utilizing a better graphical software, the AMU also recommends that forecasters use the real- 
time verification Graphical User Interface (GUI) that is currently available in the RWO. Developed by the 
AMU, this software allows forecasters to verify RAMS predictions in real-time at both surface and upper-level 
observational sites. The software incorporates the point forecast data from the current RAMS simulation and 
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RAMS version 4.3 still contains a couple drawbacks, though. First, this version still does not incorporate 
fully-digital vegetation data; however, the subsequent unreleased version 4.4 should contain this digital 
vegetation data. Second, RAMS currently has only one cumulus/precipitation parameterization scheme, the 
modified Kuo scheme (Tremback 1990), for simulations on coarse grid resolutions. The Kuo cumulus scheme 
is the most appropriate for meso-beta grid-spacing scales of 20—200 km. Meanwhile, the current operational 
configuration of RAMS at CCAFS runs the Kuo cumulus scheme on grids with 15-km and 5-km horizontal 
resolutions. At these resolutions, an alternative hybrid cumulus scheme should be utilized rather than the Kuo 
scheme (Molinari and Dudek 1992). At this time, there is no identifiable date when an additional 
cumulus/precipitation parameterization scheme will be implemented into RAMS in the near future. 
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Temperature 


Daytime surface and low-level cold bias during both seasons, peaking at -4°C to -4.5°C in the cool 
season and -2.5°C to -3.5°C in the warm season. 

Cold bias at the buoys at all hours during both seasons, again largest in the cool season (-3°C to 
-4°C). 

Temperature profile too stable in the troposphere during both seasons. 

The lowest 50 mb or so of the atmosphere are not stable enough in the early morning sounding 
during both seasons. 

Results are consistent with the 1999 warm-season evaluation in the ERDAS RAMS interim report. 
Moisture 

Nocturnal surface moist bias in the cool season of 1-2. 5°C. 

Daytime surface diy bias in warm season of -1°C to -2°C. 

Largest dew point temperature errors in mid-levels (5— 8°C in the 600— 850-mb layer) during both 
seasons. 

Wind Direction 

Rapid growth in surface wind direction errors from 20—40° in the first 2 hours of integration 
during both seasons. 

Largest errors of about 60-70° occur at the surface during the late night and early morning hours, 
both seasons. 

Wind direction errors decrease with height, most notably in the cool season where RMS errors are 
only 10-20° above 500 mb. 

Low-level easterly bias over land, both seasons (u-wind bias of -2 to -2 m s *, especially during the 
day in the warm season). 

Southerly bias at all levels above the surface during both seasons. 

Wind Speed 

Speed errors increase with height from 2.5 m s‘ l near the surface to 5-7 m s 1 at upper-levels, 
especially in the cool season. 

Positive bias near the surface in both seasons; negative speed bias above 950 mb in the cool 
season, and above 850 mb in the warm season. The magnitude of all speed biases are generally 
less than 2 m s' 1 . 

.2.2 Regime Classification Results during the 2000 Warm Season 
Surface Wind Classification 

The light wind regime experienced the largest daytime temperature RMS errors (4.5°C) and cold 
bias (-3.5°C). The wind direction errors were by far the largest during the late night and early 
morning hours (~ 80-120°) associated with this surface wind regime. 

The westerly wind regime contained the largest random temperature errors (3.5°C), wind 
direction RMS errors (75°), and u-wind component RMS errors (4.2 m s' 1 ) during the afternoon 
and evening hours. These errors are likely caused by the presence of the ECSB within the wind 


8.3 Summary of 1999-2000 Cool-Season Subjective Evaluation 

The AMU performed a frontal, precipitation, and low-level temperature inversion verification as part of the 
1999-2000 cool-season subjective evaluation. For cold frontal verification, the AMU identified observed fronts 
using GOES-8 infrared imagery and surface stations plots. The AMU subsequently verified RAMS forecast pre- 
frontal conditions, the changes in temperature, dew point temperature, and wind direction associated with the 
frontal passage, and the maximum post-frontal wind speed at seven stations along the east coast of the Florida 
peninsula. During the 1999-2000 cool season, only a few significant precipitation events occurred across the 
Florida peninsula, so the AMU qualitatively examined each event and summarized the results in a table. The 
low-level temperature inversion verification was performed for forecasts at the XMR rawinsonde. Observed and 
forecast temperature inversions were identified and verified in the lowest 3 km of the atmosphere. The results 
are summarized below. 

8.3.1 Frontal Verification 

• RAMS pre-frontal temperatures were too cold by 1.6° due to the prevailing low-level cold bias. 

• RAMS forecast fronts were generally too weak, given the fact that the average 3-h temperature and 
dew point temperature decreases associated with the frontal passages were 1.9°C and 4.6°C too 
small, respectively. 

• Forecast post-frontal maximum wind speeds were typically too weak by 2.5 ms 1 . 

8.3.2 Precipitation Verification 

• RAMS demonstrated varying skill in predicting cool-season precipitation patterns. 

• The model often missed significant pre-frontal rain bands. 

• The problems in adequately predicting frontal-associated precipitation could be caused by large- 
scale timing errors associated with the Eta model boundary conditions. 

83.3 Low-level Temperature Inversion Verification 

• RAMS predicted only about half of all low-level temperature inversions. 

• When successful, RAMS had a tendency to underestimate the magnitude of the temperature 
inversion (-2.5°C bias). 

• Many predicted inversions were not based at the surface, as observed. 

8.4 Summary of Subjective Evaluation during the 1999 and 2000 Warm Seasons 

The AMU also performed 3 separate verifications in the warm-season subjective evaluation. These three 
components include a sea-breeze, precipitation, and thunderstorm initiation verification. The sea-breeze 
verification was composed of three segments: 

• Verification of operational RAMS sea-breeze forecasts at 12 selected KSC/CCAFS wind towers 
for May- August 1999 and May-September 2000. 

• Comparison of operational 4-grid RAMS to 3-grid RAMS sea-breeze forecasts at the same 12 
KSC/CCAFS wind towers during the 2000 warm season only. 

• Benchmark of the operational RAMS to the Eta model sea-breeze forecasts at TTS for the 1999 
and 2000 warm season. 

For the instantaneous precipitation and thunderstorm initiation verifications, RAMS grid 4 was divided into 
6 zones, 3 inland and 3 coastal. In the precipitation verification, hourly forecast rain rates of 5 mm h' 1 or greater 
were verified against WSR-74C reflectivities of 32 dBZ or higher. The precipitation verification was conducted 
during the hours of 1500-2300 UTC every day from June-September 2000. 
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8.4.2 Precipitation Verification 

The precipitation verification results indicated the following about the operational RAMS forecasts: 

• The best precipitation forecasts occurred in the inland zones of grid 4 where the POD was about 
0.10-0.20 higher than the coastal zones. 

• The worst predictions for both forecast cycles occurred in zone 6 (the southeastern part of grid 4) 
where the POD was generally under 0.30. 

• The 1200 UTC cycle generally predicted precipitation better than the 0000 UTC cycle. 

• In the 1200 UTC cycle, the most accurate precipitation forecasts occurred during the hours of 
1600-2000 UTC when CSI scores (e.g. threat scores) were between 0.25 and 0.40. After 2000 
UTC, the CSI scores decreased rapidly to well under 0.20, possibly caused by the model’s inability 
to adequately forecast the evolution of observed convective outflow boundaries. 

8.4.3 Thunderstorm Initiation Verification 

The results of the thunderstorm initiation verification yielded the following conclusions: 

• The 1200 UTC forecast cycle was a much better predictor of daily thunderstorm occurrence than 
the 0000 UTC cycle. This result is expected based on the newer initial condition of the 1200 UTC 
forecast cycle compared to the typical thunderstorm initiation time. 

• The 0000 UTC forecast cycle underforecast the occurrence of thunderstorms in all grid-4 zones. 

• Thunderstorm occurrence in zone 6 (southeastern part of grid 4) was underforecast by both cycles. 

• About 50% of all RAMS forecasts correctly identified one or more zones for daily thunderstorm 
initiation. 

• About 75% of RAMS forecasts identified the timing of thunderstorm initiation to the nearest 3 
hours (± 3 h). 

The results of the precipitation and thunderstorm initiation verification both could improve by: 

• Expanding the lateral boundaries of grid 4 to displace errors resulting from boundary interactions 
further from the areas of interest. 

• Replacing the modified Kuo cumulus parameterization scheme with a hybrid cumulus scheme that 
is more appropriate for the resolutions used on RAMS grids 2 and 3(15 km and 5 km). 

• Initializing the model soil moisture with actual soil moisture observations and/or archived 
precipitation data. 

• Implementing a four dimensional data assimilation scheme that ingests high-resolution, continuous 
observational data such as WSR-88D reflectivity and radial velocity, and GOES-8 data. 

8.5 Summary of Sensitivity Experiments 

The AMU performed various sensitivity tests to isolate the possible cause(s) of the RAMS forecast errors, 
particularly the low-level cold bias. One of these experiments involved running RAMS using Eta 0-h rather than 
12-h forecasts as background fields for the initial conditions. In addition, the 0-24-h rather than the 12-36-h 
forecasts were used as boundary conditions. This experiment resulted in nearly identical forecast errors. 

An additional experiment altered the short-wave radiation scheme using the Mahrer-Pielke rather than the 
operational Chen and Cotton scheme. In the Mahrer-Pielke short-wave radiation scheme, the effects of clouds 
on short-wave radiation are ignored. In this experimental run, the surface cold bias was dramatically reduced. 
As a result of this experiment, the AMU discovered that RAMS routinely generates a widespread low-level fog 
deck across all of grid 4. The fog occurs at night over land and at all times over the ocean. The AMU has not 
identified the cause of this low-level fog problem. 
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Table A2. The status of RAMS 4-grid, 3-grid, and Eta point forecasts is displayed for January and February 2000. An 'X' denotes a completed 

forecast whereas a blank denotes a missing forecast. 
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Table A4. The status of RAMS 4-grid, 3-grid, and Eta point forecasts is displayed for June and July 2000. An 'X' denotes a completed forecast 
whereas a blank denotes a missing forecast. 
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Appendix B 

Selected Figures for the 1999-2000 Cool-season Objective Evaluation 
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1200 UTC 4*grid Cycle Wind Dir (deg) for 16.5-m towers 
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Figure B6. Vertical profiles of the temperature bias (°C) at XMR from 
the 1200 UTC operational RAMS forecast cycle during the 1999-2000 cool 
season. Biases are shown for the a) 1 1-h forecast and b) 23-h forecast. 











Appendix C 

Selected Figures for the 2000 Warm Season Objective Evaluation 
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1 200 UTC 4-grid Cycle Wind Dir (deg) for 16.5-m towers 1 200 UTC 4-grid Cycle U-Wind (in/s) for 16.5-m towers 
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1 200 UTC 4-grid Cycle 03-h Wind Dir (deg) at XMR I ?00 UTC 4-grid Cycle 22-h Wind Dir (deg) at XMR 
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(D6) 


P * p; - Sl )< i L ]= J = 0.025 , and 

P*p;-5 2 -)>/,]=l-| = 0.975, 

where P* represents probabilities calculated from the resampled distribution. Finally, the null hypothesis, H, h is 
rejected if (s, - S,)< t L or (s, - S,)> t r . The results of this resampling method are shown in Table D1 for 

most individual categorical and skill scores presented in the main report. The interpretation is as follows. Each 
column in Table DI represents a significance test by subtracting one forecast or configuration from another. 
The convention is (Forecast 1 — Forecast 2). If the test statistic is greater than 97.5%, then the skill of Forecast 
1 is significantly higher than Forecast 2. Conversely, if the test statistic is less than 2.5%, then the skill of 
Forecast 2 is significantly higher than the skill of Forecast 1 . 


Table D 1 . Levels of statistical significance (%) for various comparisons of the RAMS and Eta sea-breeze 
categorical and skill scores using a two-tailed, resampling method following Hamill (1999). The RAMS versus 
Eta tests are valid for the sea-breeze evaluation at TTS while the comparisons of different configurations and 
initializations of RAMS are valid at the 12 ICSC/CCAFS wind towers in Figure 2.2b. Scores that are statistically 
significant at 95% confidence or higher are highlighted in bold italic font. 


RAMS 1200 UTC 

RAMS 4-grid 
minus 

RAMS 4-grid 
minus 

RAMS 

minus 

RAMS 

minus 

Parameter 

minus 

RAMS 0000 UTC 

RAMS 3-grid 
(0000 UTC) 

RAMS 3-grid 
(1200 UTC) 

Eta 

(0000 UTC) 

Eta 

(1200 UTC) 

POD 

100% 

99.9% 

99.9% 

100% 

99 . 9 % 

FAR 

46.2% 

14.9% 

35.0% 

66.5% 

98.1% 

Bias 

100% 

98.5% 

99.6% 

100% 

100% 

CSI 

100% 

99 . 9 % 

99.2% 

99 . 9 % 

86.1% 

HSS 

99.6% 

99.4% 

96.3% 

99.2% 

58.3% 
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Figure El. A sample KSC/CCAFS wind-tower display from the RAMS verification graphical user 
interface developed by the AMU used to validate RAMS forecasts versus observations in real-time. This 
display shows a plot of the RAMS forecast versus observed wind speed and wind direction at towers 3, 519, and 
1012 during the 24-h forecast period from the 0000 UTC 14 May 2000 RAMS model prediction. 
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NOTICE 


Mention of a copyrighted, trademarked or proprietary product, service, or document does not constitute 
endorsement thereof by the author, ENSCO, Inc,, the AMU, the National Aeronautics and Space 
Administration, or the United States Government. Any such mention is solely for the purpose of fully informing 
the reader of the resources used to conduct the work reported herein. 
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